sourcepirate's picture
Upload folder using huggingface_hub
6f494ad verified
|
Raw History Blame Contribute Delete
3.85 kB
---
base_model: tiiuae/Falcon-H1-Tiny-R-90M
library_name: transformers
pipeline_tag: text-generation
license: other
license_name: falcon-llm-license
license_link: https://huggingface.co/tiiuae/falcon-llm-license
language:
- en
tags:
- falcon-h1
- qlora
- peft
- text-generation
- cyber-security
- gguf
datasets:
- WNT3D/Ultimate-Offensive-Red-Team
- suryanshp1/kali-linux-pentesting-data
- echel0nn1881/kimi-cyber-reasoning
- reloading0101/threat-intelligence-dataset
- Waiper/ExploitDB_DataSet
- Romit2004/LinuxCommands
- 2etatg/Debian_Hacking_Networking
---
# Falcon-H1-Tiny-R-90M — cyber-security QLoRA
A QLoRA fine-tune of [`tiiuae/Falcon-H1-Tiny-R-90M`](https://huggingface.co/tiiuae/Falcon-H1-Tiny-R-90M)
on public red-team and cyber-security datasets. Weights are merged into the base
model, so this repository loads as a normal causal language model.
## Training
| Setting | Value |
|---|---|
| Method | QLoRA (4bit base) + LoRA |
| LoRA rank / alpha / dropout | 8 / 16 / 0.05 |
| LoRA targets | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `in_proj`, `gate_proj`, `up_proj`, `down_proj` |
| Learning rate | 0.0002 |
| Epochs | 1 |
| Micro batch x grad accum | 1 x 8 |
| Max sequence length | 1024 (longer rows dropped, not truncated) |
| Optimizer | `paged_adamw_8bit` |
| Seed | 42 |
## Data
| Loader | Source | Rows (cap) |
|---|---|---|
| `redteam` | [`WNT3D/Ultimate-Offensive-Red-Team`](https://huggingface.co/datasets/WNT3D/Ultimate-Offensive-Red-Team) | 20000 |
| `kali` | [`suryanshp1/kali-linux-pentesting-data`](https://huggingface.co/datasets/suryanshp1/kali-linux-pentesting-data) | 343 |
| `kimi` | [`echel0nn1881/kimi-cyber-reasoning`](https://huggingface.co/datasets/echel0nn1881/kimi-cyber-reasoning) | 997 |
| `cti` | [`reloading0101/threat-intelligence-dataset`](https://huggingface.co/datasets/reloading0101/threat-intelligence-dataset) | 6000 |
| `exploitdb` | [`Waiper/ExploitDB_DataSet`](https://huggingface.co/datasets/Waiper/ExploitDB_DataSet) | 8000 |
| `linuxcmd` | [`Romit2004/LinuxCommands`](https://huggingface.co/datasets/Romit2004/LinuxCommands) | 6000 |
| `debian` | [`2etatg/Debian_Hacking_Networking`](https://huggingface.co/datasets/2etatg/Debian_Hacking_Networking) | 965 |
`reloading0101/threat-intelligence-dataset` is **CC-BY-NC-SA-4.0**
(non-commercial); check every source before commercial use.
## Files
- `model.safetensors` — merged weights (bf16) with the tokenizer and chat template.
- `training_config.json` — resolved configuration used for this run.
- `model-q8_0.gguf`
## Usage
Transformers:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sourcepirate/falcon-h1-tiny-cyber"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [{"role": "user", "content": "Write a tcpdump command for DNS only"}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=128)[0][inputs["input_ids"].shape[-1]:],
skip_special_tokens=True))
```
llama.cpp (CPU or GPU, no Python):
```bash
llama-cli -m model-q8_0.gguf -p "Write a tcpdump command for DNS only" --n-predict 128
```
## Evaluation
Training and per-row evaluation gate on finite loss; no benchmark scores are
claimed here. A NaN/inf loss aborts the run instead of publishing an adapter.
## Limitations and authorised use
This model is intended for **authorised** security testing, research and
education. It can produce offensive commands and may be wrong; review every
output before running it. The base model ships under the Falcon-LLM licence
([link](https://huggingface.co/tiiuae/falcon-llm-license)), and the merged model inherits it.