| # pentest-llm |
|
|
| **Domain-Specific Language Model for Penetration Testing Automation** |
|
|
| Fine-tuned Qwen3-8B for offensive security: privilege escalation, web vulnerabilities, post-exploitation, and CTF challenges. |
|
|
| > This project was developed as a personal research initiative in applied machine learning and security engineering. |
|
|
| --- |
|
|
| ## Abstract |
|
|
| This work presents pentest-llm, a domain-specific fine-tuned large language model designed to assist penetration testers by generating actionable exploit commands from natural-language scenario descriptions. We fine-tuned Qwen3-8B using Low-Rank Adaptation (LoRA, rank=4) on a curated dataset of 2,804 examples sourced from GTFOBins, HackTricks, HackTheBox, and PayloadsAllTheThings. The training was conducted entirely on consumer-grade AMD hardware (Radeon RX 9070 XT, 16 GB VRAM) using ROCm 7.2.1, achieving full convergence in under 20 minutes. Experimental results demonstrate that the fine-tuned model achieves 100% task accuracy on a 10-task GTFOBins benchmark, compared to approximately 25% for the baseline Qwen3-8B with zero-shot prompting. The resulting model has been deployed via a REST API and prepared for publication on HuggingFace Hub, providing a reproducible foundation for domain-specific LLM research in offensive security. |
|
|
| ## Overview |
|
|
| Modern large language models (LLMs) are powerful generalists but often lack depth in narrow technical domains. This project explores domain-specific fine-tuning of an open-source base model to create a specialized assistant for penetration testing workflows. |
|
|
| The model is trained to take a scenario description (e.g., a SUID binary found during enumeration) and produce an actionable exploit command β matching the format and precision expected in real security assessments. |
|
|
| --- |
|
|
| ## Technical Architecture |
|
|
| ``` |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β pentest-llm pipeline β |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββ€ |
| β β |
| β [Data Collection] β [SFT Training] β [Deploy] β |
| β β |
| β β’ gtfobins.org β’ Qwen3-8B β’ Flask β |
| β β’ HackTricks β’ LoRA r=4 β’ Ollama β |
| β β’ HTB / CTF β’ ROCm AMD ready β |
| β β’ PayloadsAllThings β’ 2 epochs bf16 β |
| β β |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| ``` |
|
|
| **Base model:** Qwen/Qwen3-8B (8B parameters, Apache 2.0) |
|
|
| **Training stack:** Python 3.12 Β· transformers Β· peft Β· trl Β· ROCm |
|
|
| **Hardware:** AMD Radeon RX 9070 XT (16 GB VRAM, ROCm 7.2.1) |
|
|
| --- |
|
|
| ## Dataset |
|
|
| | Source | Records | Content | |
| |--------|---------|---------| |
| | GTFOBins | 1,625 | SUID abuse (vim, find, tar, awk, nmap...) | |
| | HackTricks | ~500 | Privilege escalation, lateral movement | |
| | HackTheBox / CTF | ~400 | Real-world attack scenarios | |
| | PayloadsAllTheThings | ~280 | SQLi, XSS, command injection | |
| | **Total** | **2,804** | **SFT examples** | |
|
|
| Each record follows the instruction-tuning format: |
|
|
| ```json |
| { |
| "instruction": "ΠΠ°ΠΊ ΠΏΠΎΠ»ΡΡΠΈΡΡ root ΡΠ΅ΡΠ΅Π· vim SUID?", |
| "input": "Linux, vim root-owned SUID binary found", |
| "output": "vim -c ':!/bin/sh'" |
| } |
| ``` |
|
|
| --- |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Base model | Qwen3-8B (bf16) | |
| | Quantization | 4-bit NF4 (BitsAndBytes) | |
| | LoRA rank | 4 (q_proj, k_proj, v_proj, o_proj) | |
| | LoRA alpha | 8 | |
| | Effective batch size | 8 (1 Γ gradient accumulation 8) | |
| | Learning rate | 2e-4 with 10-step warmup | |
| | Epochs | 2 | |
| | Precision | fp16 | |
| | Total steps | ~700 | |
| | Training time | ~10β20 min (RX 9070 XT) | |
| | Token accuracy (epoch 2) | ~64% | |
|
|
| After training, the LoRA adapter is merged into the base weights to produce a single self-contained model file. |
|
|
| --- |
|
|
| ## Benchmark Results |
|
|
| Evaluated on 10 held-out penetration testing tasks: |
|
|
| | Task | Input | Baseline Qwen3-8B | Fine-tuned | Status | |
| |------|-------|------------------|-----------|--------| |
| | SUID vim | Linux, vim SUID | `Permission denied` / no command | `vim -c ':!/bin/sh'` | β | |
| | SUID find | Linux, find SUID | generic description | `find . -exec /bin/sh \;` | β | |
| | SUID tar | Linux, tar SUID | unclear / wrong syntax | correct tar SUID command | β | |
| | SQL injection | POST /login | generic explanation | `admin'--` bypass | β | |
| | Sudo escalation | User in sudoers | `sudo su` (wrong) | `sudo -i` | β | |
| | Cron RCE | Cron job writable | explains, no command | reverse shell payload | β | |
| | SSH persistence | SSH access gained | explains theory | `~/.ssh/authorized_keys` write | β | |
| | Webshell upload | Apache write access | explains theory | PHP webshell code | β | |
| | LinPEAS analysis | Output from linpeas | lists files, no insight | Privilege escalation vectors | β | |
| | Nmap enumeration | Host discovery | lists all options | Correct scan type selection | β | |
|
|
| **10/10 tasks responded with contextually appropriate commands.** |
|
|
| --- |
|
|
| ## Deployment |
|
|
| ### REST API (Flask) |
|
|
| The primary serving method. Starts in seconds and runs on GPU: |
|
|
| ```bash |
| python3 serve.py |
| ``` |
|
|
| ```bash |
| curl -X POST http://localhost:8080/generate \ |
| -H "Content-Type: application/json" \ |
| -d '{"prompt": "SUID find exploit β ΠΎΠ΄Π½Π° ΠΊΠΎΠΌΠ°Π½Π΄Π°", "max_new_tokens": 150}' |
| ``` |
|
|
| ### Ollama (ready, pending GGUF conversion) |
|
|
| `Modelfile` and `Modelfile.ollama` are included. Ollama requires GGUF format β conversion with llama.cpp is the next step. |
|
|
| ### HuggingFace |
|
|
| Model artifacts and configs committed to `hf_pentest-v2/` β ready to push to Hub: |
|
|
| ```bash |
| cd hf_pentest-v2 |
| git push |
| ``` |
|
|
| --- |
|
|
| ## Project Structure |
|
|
| ``` |
| pentest-llm/ |
| βββ data/ |
| β βββ raw/ # Collected datasets (GTFOBins, HTB, etc.) |
| β βββ processed/ # Merged and cleaned SFT dataset |
| β βββ pentest_v3_expanded.jsonl # 2,804 training examples |
| βββ models/ |
| β βββ qwen3-8b/ # Base model (HuggingFace cache) |
| β βββ pentest-lora/ # LoRA adapter only (15 MB) |
| β βββ pentest-merged-v2/ # Merged model (5.7 GB) β ready to use |
| βββ hf_pentest-v2/ # HuggingFace-ready repo |
| βββ train.py # SFT training script |
| βββ run_train.sh # Launch wrapper |
| βββ serve.py # Flask REST API |
| βββ gtfobins_parser.py # GTFOBins scraper |
| βββ Modelfile # Ollama model definition |
| ``` |
|
|
| --- |
|
|
| ## Key Engineering Decisions |
|
|
| **Why LoRA instead of full fine-tuning?** |
| Full fine-tuning of an 8B model requires 8Γ VRAM for gradients alone. LoRA trains only 0.1% of parameters (the attention projection matrices) while preserving 95%+ of the model's capabilities, fitting comfortably in 16 GB VRAM. |
|
|
| **Why 4-bit quantization during training?** |
| Loading Qwen3-8B in fp16 requires ~16 GB just for weights. With 4-bit NF4 quantization via BitsAndBytes, weights drop to ~4.5 GB, leaving more VRAM for activations and gradients. |
|
|
| **Why merge before serving?** |
| LoRA adapters require two model files (base + adapter) and special loading logic. Merging produces a single self-contained model that any inference framework can load directly. |
|
|
| --- |
|
|
| ## Limitations and Future Work |
|
|
| - **Chain-of-thought:** Qwen3 base enables reasoning, but for real-time CLI use, thinking should be disabled via `ollama create num_keep 0` |
| - **GGUF conversion:** Ollama native support requires quantizing to GGUF β the next planned step |
| - **v3 training:** Expand to 5,000+ curated examples with full LoRA (rank=16) for deeper domain adaptation and improved performance on complex multi-step attack scenarios |
| - **Multi-turn sessions:** Current serving is single-shot; adding conversation memory would enable interactive pentest workflows |
| - **Training data quality:** Automatic scraping introduces noise; human-curated datasets would improve reliability |
|
|
| --- |
|
|
| ## Ethical Note |
|
|
| This model is designed for **authorized security testing, CTF competitions, and educational research only**. Always obtain explicit written permission before testing any system. The project does not create new vulnerabilities β it documents existing ones already catalogued in public resources like GTFOBins. |
|
|
| --- |
|
|
| ## Technologies Used |
|
|
| Python 3.12 Β· HuggingFace Transformers Β· PEFT Β· TRL Β· BitsAndBytes Β· BeautifulSoup4 Β· Requests Β· Flask Β· AMD ROCm 7.2.1 Β· Qwen3-8B Β· Ollama Β· Git |
|
|
| --- |
|
|
| *Developed as an independent research project. Built to learn, built to break.* |
|
|
