sourcepirate's picture
Upload folder using huggingface_hub
6f494ad verified
|
Raw History Blame Contribute Delete
3.85 kB
metadata
base_model: tiiuae/Falcon-H1-Tiny-R-90M
library_name: transformers
pipeline_tag: text-generation
license: other
license_name: falcon-llm-license
license_link: https://huggingface.co/tiiuae/falcon-llm-license
language:
  - en
tags:
  - falcon-h1
  - qlora
  - peft
  - text-generation
  - cyber-security
  - gguf
datasets:
  - WNT3D/Ultimate-Offensive-Red-Team
  - suryanshp1/kali-linux-pentesting-data
  - echel0nn1881/kimi-cyber-reasoning
  - reloading0101/threat-intelligence-dataset
  - Waiper/ExploitDB_DataSet
  - Romit2004/LinuxCommands
  - 2etatg/Debian_Hacking_Networking

Falcon-H1-Tiny-R-90M — cyber-security QLoRA

A QLoRA fine-tune of tiiuae/Falcon-H1-Tiny-R-90M on public red-team and cyber-security datasets. Weights are merged into the base model, so this repository loads as a normal causal language model.

Training

Setting Value
Method QLoRA (4bit base) + LoRA
LoRA rank / alpha / dropout 8 / 16 / 0.05
LoRA targets q_proj, k_proj, v_proj, o_proj, in_proj, gate_proj, up_proj, down_proj
Learning rate 0.0002
Epochs 1
Micro batch x grad accum 1 x 8
Max sequence length 1024 (longer rows dropped, not truncated)
Optimizer paged_adamw_8bit
Seed 42

Data

reloading0101/threat-intelligence-dataset is CC-BY-NC-SA-4.0 (non-commercial); check every source before commercial use.

Files

  • model.safetensors — merged weights (bf16) with the tokenizer and chat template.
  • training_config.json — resolved configuration used for this run.
  • model-q8_0.gguf

Usage

Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sourcepirate/falcon-h1-tiny-cyber"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [{"role": "user", "content": "Write a tcpdump command for DNS only"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=128)[0][inputs["input_ids"].shape[-1]:],
                       skip_special_tokens=True))

llama.cpp (CPU or GPU, no Python):

llama-cli -m model-q8_0.gguf -p "Write a tcpdump command for DNS only" --n-predict 128

Evaluation

Training and per-row evaluation gate on finite loss; no benchmark scores are claimed here. A NaN/inf loss aborts the run instead of publishing an adapter.

Limitations and authorised use

This model is intended for authorised security testing, research and education. It can produce offensive commands and may be wrong; review every output before running it. The base model ships under the Falcon-LLM licence (link), and the merged model inherits it.