|
Download tests/STRESS_TEST.md from gesh75/multivendor-cli-configurator: direct link, hf CLI and curl.
- Browser
- Download file 4.75 kB
-
https://huggingface.co/spaces/gesh75/multivendor-cli-configurator/resolve/main/tests/STRESS_TEST.md
- Command line
-
hf download hf://spaces/gesh75/multivendor-cli-configurator/tests/STRESS_TEST.md
-
curl -L -o STRESS_TEST.md https://huggingface.co/spaces/gesh75/multivendor-cli-configurator/resolve/main/tests/STRESS_TEST.md
4.75 kB
| # v2 Feature Stress Test Report | |
| > Generated against the real **52,313-row** `commands.json` corpus | |
| > (29,509 original + 22,804 merged from the DCN tool's | |
| > `cli-export.json`, adding Microsoft/PowerShell, Linux, and Wireshark | |
| > and back-filling Arista/Huawei/Aruba/FRR/VyOS/FortiOS). | |
| > Last run: **2026-05-27** on macOS / Node v25.8 / Apple Silicon. | |
| Two harnesses cover the v2 surface: | |
| 1. **Node** — pure-JS function benchmarks (`tests/stress_test.js`). Extracts | |
| `extractPlaceholders`, `substitutePlaceholders`, `lookupConcept`, and the | |
| `CONCEPT_SYNONYMS` table directly from `index.html`, then exercises them | |
| against the production data file. No mocking, no framework, just | |
| `process.hrtime.bigint()` + `assert`. | |
| 2. **Browser** — `tests/stress_test.html`. Exercises IndexedDB round-trip and | |
| cold/warm boot via a hidden iframe loading `/index.html`. Open in the | |
| dev-server tab to run. | |
| ## How to reproduce | |
| ```bash | |
| cd /tmp/cli-work | |
| # 1. Node test suite (pure JS, no browser needed) | |
| node tests/stress_test.js | |
| # → writes tests/stress_test_results.json + exits non-zero on failure | |
| # 2. Browser test suite (IDB + boot timings) | |
| python3 -m http.server 8765 & | |
| open http://127.0.0.1:8765/tests/stress_test.html | |
| # → results render in-page and on window.__STRESS_RESULTS | |
| ``` | |
| ## Node-side results (median of 5 runs) | |
| | # | Test | Target | Actual | Pass | | |
| |---|---------------------------------------------------|------------------|-----------|------| | |
| | 3 | Filter 52,313 rows by `vendor=Cisco` | < 50 ms | **0.74 ms** | ✅ | | |
| | 4 | Filter by vendor + cat + role (3-way) | < 50 ms | **0.38 ms** | ✅ | | |
| | 5 | Free-text search (`bgp neighbor`, 2 tokens AND) | < 80 ms | **18.78 ms** | ✅ | | |
| | 6 | `lookupConcept()` against 52,313 rows | < 1500 ms total | **1384 ms** (~26 µs/row, **18,253 tagged**) | ✅ | | |
| | 6b| `lookupConcept()` correctness on canonical inputs | 10 / 10 | **10 / 10** | ✅ | | |
| | 9 | `groupQueueByVendor()` on 4-row queue | < 5 ms | **< 0.01 ms** | ✅ | | |
| |10a| `extractPlaceholders()` × 1000 | < 50 ms | **1.70 ms** | ✅ | | |
| |10b| `substitutePlaceholders()` × 1000 | < 50 ms | **0.65 ms** | ✅ | | |
| |10c| `substitutePlaceholders()` correctness | 4 / 4 | **4 / 4** | ✅ | | |
| **Result: ALL TESTS PASSED.** Raw timings in `tests/stress_test_results.json`. | |
| ## Corpus parity check | |
| All **17** vendors present in the merged `commands.json`: | |
| | Vendor | Rows | | Vendor | Rows | | |
| |-----------|-------|---|-----------|-------| | |
| | Arista | 7,434 | | FortiOS | 2,376 | | |
| | VyOS | 6,286 | | NVIDIA | 1,149 | | |
| | Cisco | 4,551 | | PAN-OS | 1,217 | | |
| | Huawei | 4,406 | | Mikrotik | 1,209 | | |
| | Microsoft | 3,352 | | Nokia | 1,053 | | |
| | FRR | 3,963 | | SONiC | 445 | | |
| | Juniper | 3,277 | | Wireshark | 2,130 | | |
| | Extreme | 2,781 | | Linux | 2,591 | | |
| | Aruba | 4,093 | | | | | |
| | **TOTAL** | **52,313 rows · 17 vendors** | | | | |
| FRR provenance: **3,963 total / 870 `live` / 1,708 `in_docs`**. The extra | |
| FRR rows came from the DCN tool's expanded scrape and are tagged neither | |
| `live` nor `in_docs` so the badge counter on the UI stays accurate. | |
| ## Browser-side targets (open `stress_test.html` to verify) | |
| | # | Test | Target | | |
| |---|---------------------------------------------------|------------------| | |
| | 1 | Cold boot — empty IndexedDB, fetch + parse 9.5 MB | < 2500 ms | | |
| | 2 | Warm boot — IDB cache hit + ETag revalidate | < 600 ms | | |
| |11 | IDB write+read round-trip on ~10 MB payload | < 600 ms | | |
| When run on the same Apple Silicon machine the iframe boot tests typically | |
| finish at **~1.2 s cold / ~0.25 s warm** and IDB at **~80 ms**, but exact | |
| numbers depend on the browser engine and disk so we don't claim a hard number | |
| here — the targets above are the gate. | |
| ## Bugs found and fixed during the stress run | |
| Two ordering bugs in `CONCEPT_SYNONYMS` surfaced from the correctness suite | |
| and were fixed in-place (substring-AND patterns are order-sensitive): | |
| 1. `"undo shutdown"` matched `iface-shutdown` before `iface-noshut` | |
| because `"shutdown"` is a substring. Fixed by moving `iface-noshut` | |
| above `iface-shutdown`. | |
| 2. `"ip route 0.0.0.0 ..."` matched `static-route` before `default-route` | |
| because `["ip route","."]` fires on any IP. Fixed by moving | |
| `default-route` above `static-route`. | |
| Both have explicit comments in `index.html` documenting the ordering | |
| constraint so future edits don't silently regress. | |