Spaces:

---

pbhappliedsystems

/

quant-eval-agent-arena

Like 2

Running on Zero

Pair request: Qwen3-1.7B Q4_K_M vs Q8_0, and SmolLM3-3B Q4_K_M vs Q8_0. Use case: small agents on modest CPUs that read untrusted text, like email, web pages or tool output. A prompt injection family would fit your setup, since each case is a clear pass or fail.

Why I ask: I ran a small CPU pilot with 40 of my own injection attacks at T=0. Qwen3-1.7B Q8_0 followed 34/40 and SmolLM3-3B Q4_K_M followed 21/40. Model and quantization change together there, so the gap cannot be attributed to quantization. The same model across Q4_K_M, Q5_K_M and Q8_0 is the comparison my data cannot make.

Cases and outputs, if useful: https://huggingface.co/datasets/guicybercode/llm-injection-pilot

@guicybercode Thanks for the request—and for publishing the pilot cases and outputs. I’d be happy to evaluate Qwen3-1.7B and SmolLM3-3B and share the resulting publication bundles with you to support your project.

Your proposed comparison is well motivated. Because the pilot changes both model and quantization simultaneously, it cannot isolate a quantization effect. quant_eval would instead evaluate each model against its own full-weight baseline across Q4_K_M, Q5_K_M, and Q8_0, using matched backends, hardware, decoding contracts, fixtures, and paired statistics. That would distinguish model differences from precision-specific behavioral changes.

A prompt-injection family is also a strong fit for the stated use case: small agents processing untrusted email, webpages, and tool output. I would be interested in assessing whether your llm-injection-pilot dataset can serve as an independent external distribution. Before using it as evidence, I would need to review its license, provenance, case construction, scoring rules, uniqueness, ambiguity, contamination boundaries, and resistance to trivial lexical cues.

I am currently completing a peer-review-driven hardening program for quant_eval and closing the known limitations documented in The quant_eval Public Corpus: Behavioral Measurement of Quantization Degradation. That work includes external-distribution validation, repeat-run controls for stochastic decoding, matched execution contracts, stronger claim lineage, and independent bundle verification. I want those controls finished before beginning a new model-family integration.

Once that work is complete, a defensible evaluation of both requested models would likely require approximately 3–5 weeks:

• prepare and provenance-lock both full-weight baselines and their Q4_K_M, Q5_K_M, and Q8_0 derivatives;
• add and validate model-specific adapters and decoding contracts;
• qualify the prompt-injection dataset and implement an auditable binary scorer;
• run calibration, ceiling/discrimination checks, and repeat-seed testing where instability warrants it;
• execute the matched production matrix;
• independently verify and package the machine-readable results and publication reports.

That estimate begins when the current hardening program is complete and assumes the upstream checkpoints, licenses, chat templates, and required inference backends are usable without model-specific blockers.

I have recorded the request as:

• Qwen3-1.7B: full-weight baseline versus Q4_K_M, Q5_K_M, and Q8_0
• SmolLM3-3B: full-weight baseline versus Q4_K_M, Q5_K_M, and Q8_0
• Primary use case: prompt-injection resistance while processing untrusted text
• Candidate external dataset: guicybercode/llm-injection-pilot

I’ll follow up when the current validation work is closed and the integration can begin.

Thanks for recording it, and 3 to 5 weeks after the hardening work is fine with me.

For your dataset review, a few things up front so you don't have to dig:

• License is MIT. All prompts were written by me for this pilot, in English and Portuguese. No cases come from third-party benchmarks.
• Case construction is in src/cases.py: 16 direct and 24 indirect attacks, plus 12 controls with no attack.
• Scoring is marker-based, in src/run.py. Two caveats are in ERROS.md: I loosened the marker match after seeing the outputs (post hoc, items 5 and 6), and a regex filter drops outputs that only quote the injection. That filter is heuristic and can err both ways. There was no systematic manual review.
• Lexical cues: the indirect payloads reuse a few markers across 8 document surfaces, so a model could learn the surface pattern. I would treat it as a small external probe, not as evidence by itself.

If any of that rules it out, a short note on what failed would help me fix it in a second version.