README.md

---
license: mit
base_model: microsoft/codebert-base
library_name: pytorch
pipeline_tag: text-classification
datasets:
  - bug-localization/BeetleBox
tags:
  - code
  - bug-localization
  - information-retrieval
  - codebert
---

# RepoTrace CodeBERT reranker

This repository contains the fine-tuned neural reranker from
[`saymanq/RepoTrace`](https://github.com/saymanq/RepoTrace). It scores an issue report against
a candidate source-code chunk. RepoTrace uses BM25 to select candidate files before applying
this model.

## Model

- Base encoder: `microsoft/codebert-base`
- Head: one linear layer over the CLS representation
- Task: cross-encoder relevance scoring for issue text and source-code chunks
- Published checkpoint: `ranker.pt`, 475.6 MiB
- Final ranker: 50% normalized BM25 score and 50% normalized CodeBERT score

The neural weight was chosen using validation MRR before the test split was evaluated.

## Training data

RepoTrace uses public issue/fix histories from
[`bug-localization/BeetleBox`](https://huggingface.co/datasets/bug-localization/BeetleBox).
Preprocessing produced 157 bugs from 13 repositories across C++, Go, Java, JavaScript, and
Python. It created 13,855 issue/chunk pairs: 2,771 positive and 11,084 negative.

Positive chunks overlap changed regions in the fixing diff. File-level fallback positives are
recorded when line-level matching is unavailable. Each positive receives two BM25 hard
negatives, one path/symbol-confusable negative, and one random negative. Files introduced by a
fix are excluded because they do not exist in the pre-fix snapshot.

Generated, vendored, minified, binary, oversized, and unsupported files are filtered. The build
records failures and counts rather than silently discarding them.

## Split strategy

The repository-disjoint split contains:

| Split | Repositories | Bugs | Pairs | Main language mix |
|---|---:|---:|---:|---|
| Train | 8 | 103 | 10,585 | C++, Go, Java, JavaScript, Python |
| Validation | 2 | 15 | 1,985 | 14 Java, 1 Python |
| Test | 3 | 39 | 1,285 | 39 Python |

No repository appears in more than one split. Every training and evaluation example uses the
repository state before the fixing commit.

## Training configuration

- Accelerator: Colab T4 CUDA GPU
- Epochs: 2
- Learning rate: `2e-5`
- Weight decay: `0.01`
- Maximum sequence length: 512 tokens
- Issue token budget: 160
- Batch size: 4
- Gradient accumulation: 8, effective batch size 32
- Loss: binary cross-entropy
- Seed: 42

Epoch 1 took 365.4 seconds and reached pair-set validation MRR `0.3053`. Epoch 2 took 370.7
seconds and reached `0.3964`, so epoch 2 was saved.

## End-to-end evaluation

BM25 retrieves the highest-scoring chunk from each of 50 distinct candidate files. CodeBERT
scores those pairs. The final system blends per-query min-max-normalized BM25 and neural scores
with neural weight `0.5`, selected on validation.

### Validation

| System | Hit@1 | Hit@5 | Hit@10 | MRR | MAP |
|---|---:|---:|---:|---:|---:|
| BM25 | 0.133 | 0.333 | 0.400 | 0.224 | 0.219 |
| Hybrid | 0.267 | 0.400 | 0.467 | 0.321 | 0.275 |

### Held-out test

| System | Hit@1 | Hit@5 | Hit@10 | Recall@10 | MRR | MAP |
|---|---:|---:|---:|---:|---:|---:|
| BM25 | 0.231 | 0.410 | 0.513 | 0.401 | 0.299 | 0.250 |
| Hybrid | 0.179 | 0.333 | 0.487 | 0.357 | 0.263 | 0.210 |

The hybrid's test MRR was 12.0% below BM25. It improved 5 individual bugs, tied 13, and
worsened 21. Candidate retrieval found at least one changed file for 29 of 39 test bugs.

The validation improvement did not transfer to the held-out Python repositories. The language
and repository composition differs sharply between validation and test; that is a plausible
source of the gap, but the current evidence does not establish causality.

See the
[`held-out error analysis`](https://github.com/saymanq/RepoTrace/blob/main/artifacts/eval/error_analysis.md)
for sampled cases.

## Serving benchmark

Batched PyTorch was faster than float ONNX Runtime and dynamic int8 ONNX Runtime in the local
CPU benchmark. At 50 candidates, median times were 2.90 seconds, 4.03 seconds, and 4.05 seconds,
respectively. The float ONNX graph matched PyTorch sample logits within `7.9e-6`.

Dynamic int8 reduced weights from about 473 MiB to 119 MiB, but its sample logit difference
reached `0.0721`. Its ranking quality has not been validated, so it is not the serving model.

## Intended use

RepoTrace suggests source files for a developer to inspect when investigating a bug in a public
repository. A human should review the issue, ranked files, and surrounding code.

## Loading the checkpoint

The checkpoint is a PyTorch state dictionary for RepoTrace's `CodeBertRanker` class. Clone the
source repository and install its ML dependencies first:

```bash
git clone https://github.com/saymanq/RepoTrace.git
cd RepoTrace
python -m pip install -e '.[ml]'
```

Then load the published weights:

```python
import torch
from huggingface_hub import hf_hub_download

from repotrace.models.codebert_ranker import MODEL_NAME, CodeBertRanker

checkpoint = hf_hub_download(
    repo_id="YOUR_USERNAME/repotrace-codebert",
    filename="ranker.pt",
)
ranker = CodeBertRanker(MODEL_NAME)
ranker.load_state_dict(torch.load(checkpoint, map_location="cpu", weights_only=True))
ranker.eval()

scores = ranker.score_pairs(
    ["S3 upload adds unexpected bytes"],
    ["FILE: services/s3/provider.py\nLANGUAGE: python\ndef put_object(...): ..."],
    device="cpu",
)
```

The returned values are uncalibrated relevance scores. Compare scores only among candidates for
the same issue.

## Out-of-scope use

- Automated code changes or unreviewed fixes
- Claims that a ranked file contains the root cause
- Security auditing or vulnerability detection
- Safety-critical decisions
- Uploading private or sensitive code to a public demo

## Known limitations

- The test set has only 39 Python bugs from Airflow, LocalStack, and Odoo.
- Validation is dominated by one large Java repository, and its selected blend did not
  generalize better than BM25 on test.
- Ten test bugs had no changed file in the 50-file candidate set.
- Vague issues and very large repositories can produce weak rankings.
- CodeBERT truncates long issue/code pairs to 512 tokens.
- Generated and vendored code is intentionally excluded.
- A live request analyzes the repository's current default branch, which can differ from the
  historical snapshot used for evaluation.
- Scores are uncalibrated relevance values, not probabilities.

## Potential biases

The data reflects public open-source projects, their issue-writing styles, naming conventions,
and contribution patterns. Proprietary repositories, smaller projects, non-English issues, and
underrepresented languages may perform differently.

## Privacy and security

- Training data comes from public repositories and issue histories.
- The public API accepts only public GitHub HTTPS URLs.
- Repository code is treated as text and never executed.
- Source code and issue bodies are not sent to third-party LLM APIs.
- Production logs exclude source contents and full issue bodies.
- Temporary snapshots are removed after each request; bare clones use a bounded cache.