README: dos configuraciones (Q8+MTP y Q6); modelo = nesi2

README.md

@@ -7,7 +7,6 @@ tags:
   - pentesting
   - security
   - uncensored
-  - qwen3
   - tool-calling
   - gguf
   - llama-cpp
@@ -21,58 +20,74 @@ pipeline_tag: text-generation
 penetration testing, vulnerability analysis, exploit development, secure
 code review, and audit automation.
 
-It is built on the **Qwen3.8-27B** architecture (same family as
-[Nesi1](https://huggingface.co/nesilabs/Nesi1)) and packaged by Nesilabs as a
-**GGUF** build for **llama.cpp**, so it runs on a single GPU without an AWQ/vLLM
-stack. This is the build we run as an autonomous **pentest agent**: it reliably
-reaches code-exec on multi-stage targets (web → WebSocket tool abuse → RCE) with
-native tool-calling.
+It is packaged by Nesilabs as a **GGUF** build for **llama.cpp**, so it runs on a
+single GPU without an AWQ/vLLM stack. This is the build we run as an autonomous
+**pentest agent**: it reliably reaches code-exec on multi-stage targets
+(web → WebSocket tool abuse → RCE) with native tool-calling.
 
 ## Highlights
 - **Unrestricted** for legitimate security use cases.
-- **27B parameters**, long context (up to **240k tokens** in production), native tool-calling.
-- **GGUF Q6_K** — single-file, runs on one 48–96 GB GPU with llama.cpp.
-- Includes the **MTP** weights (`*-mtp.gguf`) for optional speculative decode.
+- **~27B parameters**, long context (up to **240k tokens** in production), native tool-calling.
+- Two GGUF builds: **Q8_0** (≈lossless, production quality) and **Q6_K** (lighter).
+- Ships the **MTP** draft weights (`*-mtp*.gguf`) for optional speculative decode.
 - Validated end-to-end as a **pentest agent** (live RCE + loot + flag on a 3-stage CTF).
 
 ## Model details
 | | |
 |---|---|
-| Base architecture | Qwen3.8-27B |
-| Format | **GGUF** (`Nesi2-Q6_K.gguf` + `Nesi2-Q6_K-mtp.gguf`) |
-| Quantization | **Q6_K** (~22 GB weights) |
+| Format | **GGUF** (llama.cpp) |
+| Quantizations | **Q8_0** (~29 GB) · **Q6_K** (~22 GB) |
 | Context length | up to 262,144 tokens (240k used in prod) |
 | Tool-calling | OpenAI-compatible (native in llama.cpp `llama-server`) |
 | Serving | **llama.cpp `llama-server`** (OpenAI `/v1` compatible) |
 | KV cache | bf16 by default; `q8_0` to roughly halve footprint |
 
 > Note: the Hub's "Architecture" widget reads the GGUF's internal loader tag
-> (`general.architecture`), which may display as `qwen35`. That is only the
-> llama.cpp loader key baked into the file — the model is **Qwen3.8-27B**.
+> (`general.architecture`); that is only a llama.cpp loader key baked into the
+> file and does not reflect the model name. The model is **Nesi2**.
 
 ## Files
 | File | What |
 |---|---|
-| `Nesi2-Q6_K.gguf` | the model (load this one) |
-| `Nesi2-Q6_K-mtp.gguf` | MTP head for speculative decode (optional, `--model-draft`) |
+| `Nesi2-Q8_0.gguf` | the model, **Q8_0** (max quality) |
+| `Nesi2-mtp-Q8_0.gguf` | MTP draft head for Q8 (optional, `--model-draft`) |
+| `Nesi2-Q6_K.gguf` | the model, **Q6_K** (lighter) |
+| `Nesi2-Q6_K-mtp.gguf` | Q6_K with the MTP head already merged (single file) |
 
 ---
 
-## Deploy on a Vast.ai box from zero
+## Two configurations
+
+Both serve the same `nesi2` model over an OpenAI-compatible `/v1`. Pick by VRAM
+and quality target.
+
+| Config | Files | VRAM (weights + MTP) | When |
+|---|---|---|---|
+| **A — Q8_0 + MTP** | `Nesi2-Q8_0.gguf` + `Nesi2-mtp-Q8_0.gguf` | ~29 + ~3 GB | max quality, 96 GB card |
+| **B — Q6_K (+MTP)** | `Nesi2-Q6_K.gguf` (or `Nesi2-Q6_K-mtp.gguf`) | ~22 GB | lighter, more slots / smaller card |
+
+> **MTP (speculative decode)** needs a llama.cpp build that supports
+> `--spec-type draft-mtp`. It helps **single-stream latency**; under heavy
+> concurrency it did not improve aggregate throughput in our tests, so for
+> multi-user serving you can run without it and keep it for low-latency use.
+
+---
+
+## Deploy on a GPU box from zero
 
 Everything below runs **on the GPU box as root**. The gateway bits run on your
-LiteLLM host. Target image: `nvidia/cuda:12.8.1-devel-ubuntu22.04`, ≥ 160 GB disk.
+LiteLLM host. Target image: `nvidia/cuda:12.8.1-devel-ubuntu22.04`, ≥ 60 GB disk
+(≥ 90 GB if you want Q8 **and** Q6 on the same box).
 
 ### 0) Pick a GPU
-| GPU | VRAM | Fits | single-stream tok/s (Q6_K, 240k slot) |
+| GPU | VRAM | Fits | single-stream tok/s (240k slot) |
 |---|---|---|---|
-| RTX A6000 | 48 GB | 1 user @ 240k | ~22 |
+| RTX A6000 | 48 GB | 1 user @ 240k (Q6_K) | ~22 |
 | RTX PRO 6000 **Max-Q** (power-capped ~300W) | 96 GB | 1–3 users @ 240k | ~20–25 (gateway end-to-end) |
-| RTX PRO 6000 **WS** (full ~600W) | 96 GB | 1–3 users @ 240k | **~55** |
+| RTX PRO 6000 **WS** (full ~600W) | 96 GB | up to 5 users @ 240k (Q8+MTP) | **~55** |
 
-> Blackwell (PRO 6000, `sm_120`) needs CUDA toolkit **≥ 12.8**. A box whose driver
-> reports CUDA < 12.8 will fail vLLM; for llama.cpp just build with
-> `-DCMAKE_CUDA_ARCHITECTURES=120`. On Ampere (A6000) use `86`.
+> Blackwell (PRO 6000, `sm_120`) needs CUDA toolkit **≥ 12.8**; build llama.cpp
+> with `-DCMAKE_CUDA_ARCHITECTURES=120`. On Ampere (A6000) use `86`.
 
 ### 1) Deps + build llama.cpp
 ```bash
@@ -91,6 +106,9 @@ cmake --build build --config Release -j"$(nproc)" --target llama-server
 
 ### 2) Download the model
 ```bash
+# Config A — Q8 + MTP
+hf download nesilabs/Nesi2 --include "Nesi2-Q8_0.gguf" "Nesi2-mtp-Q8_0.gguf" --local-dir /root/models/nesi2
+# Config B — Q6_K
 hf download nesilabs/Nesi2 --include "Nesi2-Q6_K*.gguf" --local-dir /root/models/nesi2
 ```
 
@@ -98,35 +116,35 @@ hf download nesilabs/Nesi2 --include "Nesi2-Q6_K*.gguf" --local-dir /root/models
 Never start it with `setsid`/`nohup` over SSH (they die when the pipe closes).
 Use **tmux** so it survives the SSH session.
 
-**Profile A — single power user @ 240k** (the production default):
+#### Config A — Q8_0 + MTP, 5 users @ 240k (production, RTX PRO 6000 WS)
+`--parallel N` splits `--ctx-size` across slots → `n_ctx_slot = ctx_size / parallel`.
+For **5 slots of 240k each** you need `--ctx-size 1228800` (= 5 × 245760), a
+**quantized KV cache** (`q8_0`), and the MTP draft:
 ```bash
-GGUF=$(find /root/models/nesi2 -name "*.gguf" | grep -ivE "mmproj|vision|mtp" | sort | head -1)
 tmux new-session -d -s nesi2 "/root/llama.cpp/build/bin/llama-server \
-  --model $GGUF --alias nesi2 \
-  --ctx-size 245760 --parallel 1 --n-gpu-layers 999 -fa on \
+  --model /root/models/nesi2/Nesi2-Q8_0.gguf --alias nesi2 \
+  --n-gpu-layers 999 -fa on \
+  --cache-type-k q8_0 --cache-type-v q8_0 \
+  --spec-type draft-mtp --model-draft /root/models/nesi2/Nesi2-mtp-Q8_0.gguf --spec-draft-n-max 3 \
+  --ctx-size 1228800 --parallel 5 \
   --host 0.0.0.0 --port 8080 --api-key <YOUR_API_KEY> \
   > /root/llama.log 2>&1"
 ```
+VRAM ≈ 84 / 96 GB. Drop `--spec-type/--model-draft` if your build lacks MTP.
 
-**Profile B — 3 users (team), 240k each at the same time:**
-`--parallel N` splits `--ctx-size` across slots → `n_ctx_slot = ctx_size / parallel`.
-So for **3 slots of 240k each you need `--ctx-size 737280` (= 3 × 245760)**.
-That much KV does not fit alongside Q6_K weights in 96 GB at bf16, so use a
-**quantized KV cache** (`q8_0`, ~half the footprint, negligible quality loss):
+#### Config B — Q6_K, single power user @ 240k
 ```bash
-GGUF=$(find /root/models/nesi2 -name "*.gguf" | grep -ivE "mmproj|vision|mtp" | sort | head -1)
 tmux new-session -d -s nesi2 "/root/llama.cpp/build/bin/llama-server \
-  --model $GGUF --alias nesi2 \
-  --ctx-size 737280 --parallel 3 --n-gpu-layers 999 -fa on \
-  --cache-type-k q8_0 --cache-type-v q8_0 \
+  --model /root/models/nesi2/Nesi2-Q6_K.gguf --alias nesi2 \
+  --ctx-size 245760 --parallel 1 --n-gpu-layers 999 -fa on \
   --host 0.0.0.0 --port 8080 --api-key <YOUR_API_KEY> \
   > /root/llama.log 2>&1"
 ```
-If it still OOMs on a given card: drop to `--cache-type-k q4_0 --cache-type-v q4_0`,
-or lower per-user context (e.g. `--ctx-size 393216 --parallel 3` = 128k each),
-or move to a 2× GPU box. Under `--parallel 3` **per-stream** speed drops
-(~55 → ~20–30 tok/s/stream on a WS) but aggregate throughput goes up; this is the
-right trade for a 3-person team that isn't all generating at the exact same instant.
+For **3 users @ 240k each** on a 96 GB card, use `--ctx-size 737280 --parallel 3`
+plus `--cache-type-k q8_0 --cache-type-v q8_0`. If it OOMs, drop to
+`q4_0` KV, lower per-user context (e.g. `--ctx-size 393216 --parallel 3` = 128k
+each), or move to a 2× GPU box. Under `--parallel 3` per-stream speed drops
+(~55 → ~20–30 tok/s) but aggregate throughput goes up.
 
 Wait for load (VRAM up + `server is listening`):
 ```bash
@@ -143,8 +161,6 @@ NESI2_API_KEY=<YOUR_API_KEY>
 # then
 docker compose up -d litellm
 ```
-Get `<BOX_PUBLIC_IP>:<HOSTPORT>` with
-`vastai show instance <ID> --raw` → `public_ipaddr` and `ports["8080/tcp"][0].HostPort`.
 
 ### 5) Verify end-to-end
 ```bash
@@ -157,22 +173,18 @@ curl -s -X POST https://<your-gateway>/v1/chat/completions \
 
 ## Lessons learned & recommendations
 
-- **`reasoning_effort` breaks this chat template.** The template only accepts
-  `xhigh|medium|low`; a client sending `reasoning_effort: "high"` triggers a jinja
-  exception → HTTP 500. Strip it at the gateway (LiteLLM: `drop_params: true` +
-  `additional_drop_params: ["reasoning_effort"]`) so every client is unaffected.
 - **`--parallel 1` is mandatory to actually get the full 240k in one slot.** The
   llama.cpp default (`--parallel 4`) silently gives each slot `ctx/4`.
 - **`-fa on` (flash-attention)** — keep it on; large context is unusable without it.
 - **tmux, not setsid/nohup.** Over SSH the detached process dies and you get an
   empty log; tmux survives.
-- **Agent discipline > raw capability.** Same base as Nesi1, but as an agent Nesi2
-  shines because of *tool-calling discipline*: it acts (one concrete tool call per
-  turn) instead of narrating. Give it a lean system prompt that says: act don't
-  narrate; write your own Python for anything past a simple HTTP request
-  (WebSocket, sockets, rotating CSRF); get code-exec first and confirm with `id`;
-  and **keep durable state on disk** (a `NOTES.md` + reusable scripts) so a context
-  compaction doesn't make it re-derive what it already solved.
+- **Agent discipline > raw capability.** As an agent Nesi2 shines because of
+  *tool-calling discipline*: it acts (one concrete tool call per turn) instead of
+  narrating. Give it a lean system prompt that says: act don't narrate; write your
+  own Python for anything past a simple HTTP request (WebSocket, sockets, rotating
+  CSRF); get code-exec first and confirm with `id`; and **keep durable state on
+  disk** (a `NOTES.md` + reusable scripts) so a context compaction doesn't make it
+  re-derive what it already solved.
 - **Isolate the agent's filesystem for honest evals.** A harness whose shell can
   read your machine will "solve" a CTF by reading the solution/writeups off disk
   instead of attacking live. Deny reads outside the target (regex guard in the tool
@@ -180,13 +192,13 @@ curl -s -X POST https://<your-gateway>/v1/chat/completions \
   stays open.
 - **KV quantization scales users cheaply.** `--cache-type-k/v q8_0` roughly doubles
   how many 240k slots fit, with no quality drift we could measure; it's what makes
-  the 3-user profile fit a single 96 GB card.
+  the multi-user profiles fit a single 96 GB card.
 - **Max-Q vs WS.** The power-capped Max-Q PRO 6000 is ~2.5× slower than the full WS
   for ~0.25 $/h less. For interactive/agent use pay for the WS; for cheap batch,
   Max-Q is fine.
-- **Speculative decode (MTP).** The `*-mtp.gguf` draft head is included; under
-  concurrency it did **not** help throughput in our tests (same finding as Nesi1's
-  NEXTN), so we run without it and keep it available for single-stream latency.
+- **Speculative decode (MTP).** The `*-mtp*.gguf` draft head is included; under
+  concurrency it did **not** help throughput in our tests, so we run without it for
+  multi-user serving and keep it available for single-stream latency.
 
 ## ⚠️ Legal Notice / Disclaimer
 
@@ -214,5 +226,4 @@ By using Nesi2 you accept this notice and the Nesilabs Terms of Service and
 Acceptable Use Policy.
 
 ## License
-Released under **Apache 2.0**. Based on the Qwen3.8-27B architecture.
-Adapted and packaged by Nesilabs.
+Released under **Apache 2.0**. Packaged and optimized by Nesilabs.