--- base_model: tiiuae/Falcon-H1-Tiny-R-90M library_name: transformers pipeline_tag: text-generation license: other license_name: falcon-llm-license license_link: https://huggingface.co/tiiuae/falcon-llm-license language: - en tags: - falcon-h1 - qlora - peft - text-generation - cyber-security - gguf datasets: - WNT3D/Ultimate-Offensive-Red-Team - suryanshp1/kali-linux-pentesting-data - echel0nn1881/kimi-cyber-reasoning - reloading0101/threat-intelligence-dataset - Waiper/ExploitDB_DataSet - Romit2004/LinuxCommands - 2etatg/Debian_Hacking_Networking --- # Falcon-H1-Tiny-R-90M — cyber-security QLoRA A QLoRA fine-tune of [`tiiuae/Falcon-H1-Tiny-R-90M`](https://huggingface.co/tiiuae/Falcon-H1-Tiny-R-90M) on public red-team and cyber-security datasets. Weights are merged into the base model, so this repository loads as a normal causal language model. ## Training | Setting | Value | |---|---| | Method | QLoRA (4bit base) + LoRA | | LoRA rank / alpha / dropout | 8 / 16 / 0.05 | | LoRA targets | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `in_proj`, `gate_proj`, `up_proj`, `down_proj` | | Learning rate | 0.0002 | | Epochs | 1 | | Micro batch x grad accum | 1 x 8 | | Max sequence length | 1024 (longer rows dropped, not truncated) | | Optimizer | `paged_adamw_8bit` | | Seed | 42 | ## Data | Loader | Source | Rows (cap) | |---|---|---| | `redteam` | [`WNT3D/Ultimate-Offensive-Red-Team`](https://huggingface.co/datasets/WNT3D/Ultimate-Offensive-Red-Team) | 20000 | | `kali` | [`suryanshp1/kali-linux-pentesting-data`](https://huggingface.co/datasets/suryanshp1/kali-linux-pentesting-data) | 343 | | `kimi` | [`echel0nn1881/kimi-cyber-reasoning`](https://huggingface.co/datasets/echel0nn1881/kimi-cyber-reasoning) | 997 | | `cti` | [`reloading0101/threat-intelligence-dataset`](https://huggingface.co/datasets/reloading0101/threat-intelligence-dataset) | 6000 | | `exploitdb` | [`Waiper/ExploitDB_DataSet`](https://huggingface.co/datasets/Waiper/ExploitDB_DataSet) | 8000 | | `linuxcmd` | [`Romit2004/LinuxCommands`](https://huggingface.co/datasets/Romit2004/LinuxCommands) | 6000 | | `debian` | [`2etatg/Debian_Hacking_Networking`](https://huggingface.co/datasets/2etatg/Debian_Hacking_Networking) | 965 | `reloading0101/threat-intelligence-dataset` is **CC-BY-NC-SA-4.0** (non-commercial); check every source before commercial use. ## Files - `model.safetensors` — merged weights (bf16) with the tokenizer and chat template. - `training_config.json` — resolved configuration used for this run. - `model-q8_0.gguf` ## Usage Transformers: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "sourcepirate/falcon-h1-tiny-cyber" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") messages = [{"role": "user", "content": "Write a tcpdump command for DNS only"}] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=True ).to(model.device) print(tokenizer.decode(model.generate(**inputs, max_new_tokens=128)[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ``` llama.cpp (CPU or GPU, no Python): ```bash llama-cli -m model-q8_0.gguf -p "Write a tcpdump command for DNS only" --n-predict 128 ``` ## Evaluation Training and per-row evaluation gate on finite loss; no benchmark scores are claimed here. A NaN/inf loss aborts the run instead of publishing an adapter. ## Limitations and authorised use This model is intended for **authorised** security testing, research and education. It can produce offensive commands and may be wrong; review every output before running it. The base model ships under the Falcon-LLM licence ([link](https://huggingface.co/tiiuae/falcon-llm-license)), and the merged model inherits it.