--- base_model: microsoft/codebert-base tags: - codebert - linevul - vulnerability-detection - research --- # LineVul-MoE A research release of 110 fine-tuned CodeBERT checkpoints: 11 model variants, five training seeds, and two fixed BigVul splits. The base encoder is [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base). The original [LineVul](https://github.com/awsm-research/LineVul) is the reference for the first-layer attention scoring method. ## Checkpoints - **All-in-One**: binary vulnerability detection over the full training split. - **Exclude**: one classifier per target CWE, trained on its vulnerable functions and all non-vulnerable functions. Vulnerable functions outside the target CWE are excluded. - **One-Hot**: one classifier per target CWE, initialized from the corresponding All-in-One checkpoint. The target CWE is positive and the other four top CWE classes are negative. Non-vulnerable functions are outside this training task. The five target classes are CWE-119, CWE-20, CWE-399, CWE-125, and CWE-200. The training seeds are 42, 2026, 31415, 27182, and 123456. The `official` split preserves the published LineVul split. The `project` split groups projects, CVEs, commits, and matching function hashes before splitting. Exact prepared split fingerprints are recorded in `reproduction.json`. ## Download a single checkpoint ```python from huggingface_hub import hf_hub_download from safetensors.torch import load_file path = hf_hub_download( repo_id="/LineVul-MoE", revision="", subfolder="checkpoints/LineVul-MoE_Exclude_official_CWE-119_seed42", filename="model.safetensors", ) state = load_file(path, device="cpu") # Build the original binary classifier with src/models.py in the experiment code. # Then use model.load_state_dict(state, strict=True) and model.eval(). ``` These files contain the original encoder and one-logit classification head tensors. Load them with the experiment's `src/models.py` model definition. Use `manifest.csv` to select a checkpoint and verify its SHA-256. ## Reproduce inference and evaluation **Experiment code on GitHub:** repository URL and fixed commit pending. The upstream LineVul repository above describes the reference method. 1. Reconstruct the fixed split and tokenize with the original CodeBERT tokenizer and 512-token input window. The fingerprints in `reproduction.json` identify the source run configuration, tokenizer, and prepared split files. 2. For function-level inference, use the binary model's logit and apply `sigmoid(logit / temperature)`. Use the validation-derived threshold stored in `reproduction.json` to reproduce the binary prediction. The One-Hot classifier estimates the target CWE among vulnerable Top-5 functions. 3. For line-level ranking, reproduce the original evaluator: sum first-layer attention across heads and query positions, zero the beginning and ending special-token scores, and aggregate token scores by source line. The reference evaluator includes padded query positions in this aggregation. Evaluate only the designated Top-5 vulnerable functions with a verified flaw line inside the input window. Compute the per-function metrics and average over the reported population and training seeds. ## Distribution terms The [upstream CodeBERT code repository](https://github.com/microsoft/CodeBERT/blob/master/LICENSE) carries an MIT license. The license for distribution of this fine-tuned checkpoint collection remains to be confirmed before public release.