dilekmandal commited on
Commit
cdcfe5e
·
verified ·
1 Parent(s): d7acf82

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +18 -15
README.md CHANGED
@@ -194,28 +194,31 @@ The model was fine-tuned using LoRA. Detailed hyperparameters, training duration
194
 
195
  ### Testing Data
196
 
197
- The reported evaluation used the test split of the dataset linked above. Since code overlaps and shared code families were identified across splits, the test set should not be treated as fully independent.
198
 
199
- ### Metrics
200
 
201
- Relevant metrics for this classification task include:
202
- - Accuracy
203
- - Precision
204
- - Recall
205
- - F1 score
206
 
207
- CWE classification should also be evaluated separately to determine whether the model identifies vulnerability types correctly.
208
 
209
- ### Results
210
 
211
- Previously recorded local evaluation results for this experimental adapter were:
212
 
213
- - **Accuracy:** 99.95%
214
- - **Precision:** 100%
215
- - **Recall:** 99.92%
216
- - **F1 score:** 99.96%
217
 
218
- These results are preliminary and may be inflated by cross-split code overlap and shared code families. They should not be interpreted as expected performance on unseen, real-world Java applications. Independent evaluation with leakage-resistant splits is needed.
 
 
 
 
 
 
 
 
 
 
219
 
220
  ## Technical Specifications
221
 
 
194
 
195
  ### Testing Data
196
 
197
+ The initial evaluation used the test split of the prepared dataset. During dataset auditing, exact normalized-code overlaps and shared code families were identified across the training, validation, and test splits.
198
 
199
+ Consequently, the initial test results may overestimate the model's generalization performance.
200
 
201
+ ### Evaluation Metrics
 
 
 
 
202
 
203
+ The following metrics are relevant for evaluating this classification task:
204
 
205
+ Accuracy
206
 
207
+ Precision
208
 
209
+ Recall
 
 
 
210
 
211
+ F1 score
212
+
213
+ CWE-level evaluation is also necessary to assess whether the model correctly identifies specific vulnerability types, rather than only predicting whether a code sample is vulnerable.
214
+
215
+ ### Evaluation Status
216
+
217
+ Initial local evaluation experiments have been conducted. However, due to cross-split code overlap and shared code families, the existing results are not reported here as reliable estimates of generalization performance.
218
+
219
+ Further evaluation is needed using leakage-resistant, code-family-based data splits and an independent test set. CWE-level error analysis is also planned to better understand the model's strengths and limitations.
220
+
221
+ The model should be considered an experimental research artifact, not a validated security analysis tool.
222
 
223
  ## Technical Specifications
224