| --- |
| license: other |
| license_name: lfm1.0 |
| license_link: LICENSE |
| base_model: LiquidAI/LFM2.5-Encoder-350M |
| pipeline_tag: text-classification |
| library_name: onnx |
| tags: |
| - security |
| - code |
| - cwe |
| - vulnerability-detection |
| - lfm2 |
| - onnx |
| - webgpu |
| --- |
| |
| # secjev encoder |
|
|
| [LFM2.5-Encoder-350M](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M) fine-tuned to |
| answer the 25 questions of the CWE Top 25 about a window of up to 240 lines of source code. |
| One forward pass reads a window and returns P(yes) for all 25 questions, for example |
| "Is a SQL query built here by concatenating or interpolating data, rather than by binding |
| parameters?" (`cwe_89`). It reads 28 languages, from C and Go to PHP and Solidity. |
|
|
| ``` |
| window text ──► LFM2.5 bidirectional encoder ──► attention pool (one per question) ──► linear ──► sigmoid(logit / T) |
| ``` |
|
|
| It is a triage aid that ranks where to look first. It does not prove that a flaw exists. |
|
|
| ## Files |
|
|
| | path | what | |
| |---|---| |
| | `model/model.safetensors`, `model/secjev.json` | PyTorch weights (1.4 GB): the encoder plus the per-question attention pool and the linear head; labels, questions, temperature | |
| | `onnx/model.onnx` | fp32 ONNX graph (1.4 GB): matches PyTorch to 1e-5 in the logits | |
| | `onnx/model_q8.onnx` + `onnx/model_q8.weights` | 8-bit weights in blocks of 32 with fp32 maths (593 MB), for ONNX Runtime on CPU or CUDA, and WebGPU | |
| | `onnx/model_q8_wasm.onnx` | the same weights file, with attention written as per-head operators, for onnxruntime-web on WebAssembly | |
| | `onnx/bundle.json` | labels, questions, per-language limits, temperature, window rules and the file-extension map | |
| | `*/tokenizer.json` | the base model's tokenizer | |
| | `onnx/validate-validation.json` | the validation numbers below | |
|
|
| Each ONNX graph takes `input_ids` (int64, `[1, n]`, n ≤ 8192, BOS first, no padding) for |
| one window and returns `logits` (`[1, 25]`). P(yes) is `sigmoid(logit / temperature)`. |
| Score windows one at a time, because the graphs use ONNX Runtime's `MultiHeadAttention` |
| without a padding mask. That keeps an 8,192-token window from building a 4.3 GB score |
| matrix. The two 8-bit graphs share `model_q8.weights`, so download it next to whichever |
| graph you use. |
|
|
| ## Input format |
|
|
| The model was trained on windows of this exact shape: a header, then each line numbered |
| and cut at 400 characters. Files are tiled every 240 lines from line 1, and a tile whose |
| text is over 33,084 characters is halved until it fits. `bundle.json` → `window` holds |
| these constants. `limits` says which languages each question applies to; the memory-safety |
| questions, for example, are asked only of C and C++. |
|
|
| ``` |
| // ==== src/app.py:1-240 of 512 ==== |
| 1 | import os, subprocess |
| 2 | from flask import request |
| ... |
| ``` |
|
|
| ## Usage (ONNX Runtime, Python) |
|
|
| ```python |
| import json |
| import numpy as np |
| import onnxruntime as ort |
| from tokenizers import Tokenizer |
| |
| d = "onnx/" |
| meta = json.load(open(d + "bundle.json")) |
| tok = Tokenizer.from_file(d + "tokenizer.json") |
| sess = ort.InferenceSession(d + "model_q8.onnx", providers=["CPUExecutionProvider"]) |
| |
| def window(path, lines, start, total): # the format the model was trained on |
| body = "\n".join(f"{start + i} | {l[:400] + ' ... [long line cut]' if len(l) > 400 else l}" |
| for i, l in enumerate(lines)) |
| return f"// ==== {path}:{start}-{start + len(lines) - 1} of {total} ====\n" + body |
| |
| src = open("app.py").read().split("\n")[:240] |
| ids = tok.encode(window("app.py", src, 1, len(src))).ids[: meta["max_length"]] |
| logits = sess.run(None, {"input_ids": np.array([ids], dtype=np.int64)})[0][0] |
| p = 1 / (1 + np.exp(-logits / meta["temperature"])) |
| for i in np.argsort(-p)[:5]: |
| cwe = meta["labels"][i] |
| print(f"{p[i]:.3f} {cwe} {meta['questions'][cwe]}") |
| ``` |
|
|
| In a browser, load the graphs through the `onnxruntime-web/webgpu` entry, which runs 8-bit |
| `MatMulNBits`. The package's default entry does not. Use `model_q8_wasm.onnx` when WebGPU |
| is missing. Multi-threaded WebAssembly needs COOP/COEP headers. |
|
|
| ## Training |
|
|
| - **Real labels only.** Positives are the window before a CVE/GHSA fix, labelled on the |
| questions its advisory names. The same place after the fix is the matching negative. Other |
| questions on fix windows are masked. Windows from a sweep of ordinary open-source code are |
| negatives on every question, at weight 0.25. No LLM-generated labels are used. |
| - **Pair loss.** Within each fix, the window before the fix should score above the window |
| after it (a logistic loss on the gap). This term made the model learn the flaw itself |
| rather than "looks like code that gets fixed". |
| - **Attention pooling.** Each question learns its own weighting of the window's tokens. The |
| head learns at 5e-4 and the encoder at 3e-5, in bf16, up to 8,192 tokens. |
| - **Selection and calibration.** The checkpoint was chosen by `auroc` on the fix validation |
| windows. One temperature (1.6356) was fitted on held-out calibration windows. |
| - **Data.** The permissively licensed part of the secjev dataset: 52,673 fix windows and |
| 200,000 ordinary windows. Each fix window is seen twice per epoch. |
|
|
| ## Evaluation |
|
|
| Validation split: 3,060 fix windows plus 10,000 ordinary windows, 209,900 |
| (window, question) scores in all. |
|
|
| | | auroc (before vs after fix) | pre_above_post | before fix vs ordinary | fixed vs ordinary | ordinary flagged (p ≥ 0.5) | |
| |---|---|---|---|---|---| |
| | PyTorch, bf16 | 0.5830 | 0.6560 | 0.9369 | 0.8996 | 0.75% | |
| | `model.onnx` (fp32) | 0.5832 | 0.6680 | 0.9368 | 0.8994 | 0.75% | |
| | `model_q8.onnx` | 0.5830 | 0.6691 | 0.9370 | 0.8999 | 0.76% | |
| | `model_q8_wasm.onnx` | 0.5830 | 0.6674 | 0.9370 | 0.8999 | 0.76% | |
|
|
| What the columns mean: |
|
|
| - **before fix vs ordinary (≈0.94):** the model separates code that later needed a security |
| fix from ordinary code well. |
| - **auroc and pre_above_post:** it separates the vulnerable version from its own fixed |
| version, often a few changed lines, only modestly. |
| - **fixed vs ordinary (0.90):** much of the signal is "this is the kind of code where flaws |
| of this class live". Read a high score as "look here", not "this is vulnerable". |
|
|
| Against fp32, the 8-bit graphs move a probability by 0.0005 on average (99th percentile |
| 0.009, max 0.056). They keep 98.4% of the top 500 scores and 99.97% of the p ≥ 0.5 calls. |
|
|
| Speed per window, q8 (RTX 4070 Ti SUPER, i9-14900KF): |
|
|
| | tokens | CUDA | CPU, 32 threads | WebGPU (Chrome) | WebAssembly (Chrome) | |
| |---|---|---|---|---| |
| | 2,048 | 0.05 s | 1.3 s | 0.17-0.33 s | 3.0 s | |
| | 8,192 | 0.3 s | 5.9 s | 1.5 s | 15 s | |
|
|
| ## Limitations |
|
|
| - It sees one window and nothing else: no callers, no configuration, no data flow across |
| files. Questions such as authorization (`cwe_862`) or CSRF (`cwe_352`) often hinge on |
| context outside the window. |
| - Most training positives come from public CVE fixes, so the scores lean toward flaws that |
| were found and fixed in open-source projects. |
| - Expect false positives on ordinary code at roughly the rate in the table. Treat the |
| output as a ranking for human review. |
|
|
| ## License |
|
|
| This model is a derivative of LiquidAI/LFM2.5-Encoder-350M and is distributed under the |
| [LFM Open License v1.0](LICENSE). Commercial use is licensed only to organisations below |
| US$10M in annual revenue; see section 5 of the license. |
|
|
| Changes from the base model: the encoder weights were fine-tuned, a per-question attention |
| pool and a 25-way linear head were added, and the result was exported to ONNX, including an |
| 8-bit quantised copy. |
|
|
