netops-intent2cli-qwen0.5b-lora — v2

English operator-style intent → vendor CLI, with measured limits. Experimental, not production-safe.

Treat every output as an untrusted suggestion. Review vendor release documentation and test in a lab before use. Never execute generated changes autonomously.

Evaluation

Greedy decoding, 128 new-token cap, the unchanged v1 system/user prompt. The 108 v1 test examples remain unchanged and their commands occur in neither training nor validation. Exact match strips surrounding whitespace only. Normalized match also case-folds, collapses whitespace, and removes quotes/code fences; it is not a semantic correctness score.

model	original v1 test EM	original v1 test norm	operator test EM	operator test norm
base	0/108 (0.0%)	0/108 (0.0%)	0/88 (0.0%)	0/88 (0.0%)
v1	10/108 (9.3%)	10/108 (9.3%)	2/88 (2.3%)	3/88 (3.4%)
v2	12/108 (11.1%)	12/108 (11.1%)	9/88 (10.2%)	9/88 (10.2%)

Original v1 test: per vendor

vendor	n	base EM/NM	v1 EM/NM	v2 EM/NM
Arista	9	0/0	0/0	0/0
Aruba	6	0/0	1/1	1/1
Cisco	6	0/0	1/1	1/1
Extreme	6	0/0	0/0	1/1
FRR	5	0/0	0/0	0/0
FortiOS	1	0/0	0/0	0/0
Huawei	5	0/0	1/1	1/1
Juniper	9	0/0	0/0	0/0
Linux	8	0/0	2/2	2/2
Microsoft	12	0/0	1/1	2/2
Mikrotik	7	0/0	0/0	0/0
NVIDIA	6	0/0	1/1	1/1
Nokia	6	0/0	0/0	0/0
PAN-OS	2	0/0	0/0	0/0
SONiC	5	0/0	0/0	0/0
VyOS	4	0/0	0/0	0/0
Wireshark	11	0/0	3/3	3/3

Synthetic operator-style test: per vendor

vendor	n	base EM/NM	v1 EM/NM	v2 EM/NM
Arista	8	0/0	0/0	0/0
Aruba	2	0/0	0/0	0/0
Cisco	6	0/0	0/0	0/0
Extreme	5	0/0	0/0	1/1
FRR	4	0/0	0/0	0/0
Huawei	2	0/0	0/0	0/0
Juniper	7	0/0	0/0	0/0
Linux	7	0/0	0/0	1/1
Microsoft	11	0/0	1/2	3/3
Mikrotik	5	0/0	0/0	0/0
NVIDIA	6	0/0	0/0	1/1
Nokia	5	0/0	0/0	0/0
PAN-OS	2	0/0	0/0	0/0
SONiC	4	0/0	0/0	0/0
VyOS	3	0/0	0/0	0/0
Wireshark	11	0/0	1/1	3/3

Data and methodology

v1: 565 train / 102 validation / 108 test, unchanged command-disjoint split.
v2 training: 1215 examples, adding 650 new synthetic operator-style requests across 17 vendors and 541 distinct corpus command labels. Of these, 525 are new phrasings of v1 training labels; there are 94 genuinely new command labels, and 650 unique commands in the full v2 training split. FortiOS adds 57 requests across six configuration families.
New operator-style held-out slice: 88 requests for original test commands. Only commands present with the matching vendor/OS in the 70,006-row source Parquet are eligible; credential-related examples are excluded. It is a subset, not a replacement for the frozen 108.
All 738 new train/test pairs use command strings copied verbatim from multivendor-cli-commands. New intents are synthetic, generated offline with assistant-authored semantic rules and paraphrase grammar. No vendor documentation prose is copied. No xAI requests were made for v2.
New data passed exact corpus membership, matching vendor/OS membership, whitespace-normalised command split leakage checks, surface near-duplicate rejection, firewall action-polarity checks, and restricted-work-term scans (zero hits). This is not exhaustive semantic validation.
v1's inherited curated examples remain. Sixteen original test commands are curated rather than exact source-corpus rows. Frozen benchmark provenance is preserved rather than silently rewriting labels.
Validation alone selected the checkpoint; the two test sets were evaluated after selection. This single run does not establish statistical significance or state-of-the-art performance.

Training

Continued from the v1 adapter (3 prior epochs), not a fresh base. LoRA rank 16, alpha 32, dropout .05, q/k/v/o projections; 3 extra epochs run, checkpoint selected at extra epoch 1. Rank 32/all-linear and 1.5B were not tested: the measured CPU budget and memory pressure did not support a fair additional run.

AdamW lr 1e-4 with warmup/linear decay, completion-only loss, fp32, CPU, 7 torch threads, process nice +15. Effective batch 8 via microbatch 2 and accumulation 4; the initial batch-8 attempt exceeded memory, so the final run restarted cleanly from v1. Seed 864. Early stopping patience 2 on validation loss with minimum improvement .002. Final training time 1460 s. Best validation loss 1.118536.

extra epoch	training loss	validation loss	elapsed seconds
initial v1	—	1.142961	0
1	1.128728	1.118536	500
2	0.766017	1.144932	981
3	0.525927	1.183348	1460

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(base),
    "gesh75/netops-intent2cli-qwen0.5b-lora",
)
messages = [
    {"role": "system", "content": "You are a network CLI assistant. Reply with only the exact CLI command(s) for the given vendor, no explanation."},
    {"role": "user", "content": "Vendor: Juniper (junos)\nIntent: need a compact interface and addressing overview"},
]
x = tok.apply_chat_template(messages, tokenize=True, return_dict=True,
                            add_generation_prompt=True, return_tensors="pt")
y = model.generate(**x, max_new_tokens=128, do_sample=False)
print(tok.decode(y[0, x["input_ids"].shape[1]:], skip_special_tokens=True))

The adapter alone is the open-vocabulary default. An optional lookup mode over the command base is described in Retrieval + validator pipeline.

Limitations

These are synthetic operator-style paraphrases, not independent human shift conversations. Grammar templates and numeric variants remain; near-duplicate filtering is heuristic. Shared authoring conventions between train and operator test can overestimate generalisation.
Commands are attested in a public corpus, not necessarily valid or safe for a device/release. The corpus includes placeholders, incomplete syntax productions and questionable entries. Some original intents omit values found in the label, so exact-match failure need not mean the underlying task was impossible to understand.
Original test has only one FortiOS example, which is curated; operator test has no FortiOS. Extra training coverage does not establish FortiOS reliability. Rare vendor counts are too small for broad claims.
The model still hallucinates keywords/paths and may copy fixture values, use unsafe defaults, or return the wrong operation. No live device tests or production validation were performed.
No claims of 94% accuracy, best-in-class status, or autonomous change safety.

Retrieval + validator pipeline (optional lookup mode)

Plain v2 — the Usage example above — remains the open-vocabulary default. Use it when the command may not already be in the base. The pipeline below is an optional lookup mode. The v2 adapter weights are unchanged.

The adapter alone memorizes little: 12/108 exact on the frozen test and 9/88 on the operator-style test. Put the command base around it and the same adapter gets much further on commands that exist in gesh75/multivendor-cli-commands:

Retrieve the 5 nearest commands for the vendor/OS from the base (BM25 + char n-gram TF-IDF + MiniLM embeddings, rank-fused).
Model (this adapter) sees the candidates in the prompt, picks one and fills values.
Validate: the output must match the shape of a real base command for that vendor (numbers, IPs and base placeholders are wildcards). If not, return the best retrieved command with values filled from the intent.

Greedy decoding, same frozen sets, exact match (normalized match is identical in every row here):

Variant	Frozen test (108)	Operator test (88)	Verified outputs (108 / 88)
base Qwen2.5-0.5B	0	0	1 / 0
v1 adapter	10	2	14 / 6
v2 adapter	12	9	23 / 19
retrieval only (top-1 filled, no model)	61	54	108 / 88
v2 + retrieval in prompt	30	30	48 / 43
v2 + retrieval + validator	63	58	108 / 88

Read this before trusting the table.

The gold commands are in the base, so retrieval can find them. The numbers measure "find the right entry and fill values", not "write a command nobody has seen". If the target command is not in the base the pipeline cannot return it: with all test commands (and their templates) removed from the base, retrieval scores 0, and the validator fallback lowers v2 from 12 to 2 on the frozen test, because it replaces novel but correct output with a base command.
Most of the gain is retrieval. The model adds about +2 (frozen) and +4 (operator) over retrieval alone, on sets this small that is within noise.
Verified means the command's shape exists in the base. It does not mean it is correct for your intent or safe on your device. 15 of the 196 gold commands are hand-curated and not in the base, so the validator would reject them.
Intents are synthetic paraphrases, not real operator tickets. Review every command before running it.

Run it from this repo: python rag/infer_rag.py --vendor Arista --os eos --intent "show IPv6 BGP peers with AS path regex" (needs torch transformers peft pyarrow rank_bm25 scikit-learn sentence-transformers huggingface_hub, MiniLM, and the gesh75/multivendor-cli-commands parquet, which is downloaded on first run; add --no-model for retrieval only). A retrieval-only browser demo (no model) is in the Space: intent-finder.html. See rag/README.md.

License

Apache-2.0 for the adapter, matching the base model. The dataset retains its existing MIT license; vendor names/syntax are factual identifiers and trademarks of their owners.

GESH · NetOps tools on Hugging Face


📡 netlog-ai	Sanitize-first network log analysis — browser lite demo of netlog-ai (PyPI 2.1.0)
🛠️ multivendor-cli-configurator	70k+ CLI commands across 17 vendors — cheatsheet, N-vendor compare, CLI Studio, Intent finder
🗂️ multivendor-cli-commands	The same 70,006 commands as a Parquet dataset
🧭 netops-intent2cli	v1 + v2 synthetic operator intents → corpus command pairs
🤖 netops-intent2cli-qwen0.5b-lora	LoRA adapter: English intent → vendor CLI (experimental)