Binamra00/refactoring-forecaster

Refactoring forecaster (code, history and discourse)

This model forecasts whether developers refactor a Java production file before the next release. It reads three views of the file, all written before the release: its code (the file's skeleton and its methods), its history (each commit that changed it: the message and the file's changed lines) and its discourse (the bodies, comments and reviews of the issues and pull requests linked to those commits). It returns a probability.

It is the model of A Language Model That Reads Code, History and Discourse, Chapter 5 of Binamra Aryal's MS thesis at The University of Texas at Arlington.

Training data	10.5281/zenodo.24117444 (version 2.1), the training releases of five Java repos
Code and notebooks	refactoring-forecast-replication
Mined inputs	10.5281/zenodo.24182007

The model

Part	What it is
Code encoder	microsoft/codebert-base, the first token's vector of each window
History encoder	a second copy of microsoft/codebert-base, the first token's vector
Discourse encoder	sentence-transformers/all-mpnet-base-v2, the mean of the token vectors
Windows	510 tokens without overlap (code, history); 382 tokens, each starting halfway through the last (discourse); at most 64 windows per text
Text budget	each view of a row reads at most 64 windows: the skeleton first, then the methods largest first; the newest commits; the newest discourse texts
Per text	the mean of its window vectors
Per view	attention pooling over the view's texts; a learned "empty" vector when a row has no text in a view
Output	the three view vectors joined (2,304 numbers), dropout 0.2, one linear layer, sigmoid

All weights, the encoders' included, were fine-tuned together on the labels: AdamW, learning rate 2e-5 (encoders) and 1e-3 (new layers), 16 rows per step, bfloat16, on one A100. Each epoch used every refactored training row and as many unrefactored rows of the same repo, drawn anew. Training stopped after 7 epochs; these are the weights of epoch 4, the epoch with the highest validation AP (0.106).

How well it forecasts

On the test releases (200,438 rows, 3.2% refactored), at the decision threshold of 0.794 chosen on the validation releases:

Precision	Recall	F1	ROC-AUC	AP
0.095	0.297	0.144	0.713	0.079

It ranks refactored files above the rest better than chance in every repo, with an AP 2.0 to 3.4 times the repo's base rate, but only about one flag in ten is right. Use it to order files for review, not to decide alone.

Using it

The weights fit the model classes of the replication package's Notebook 4. forecaster.py holds those classes and a small loader:

# pip install torch transformers safetensors huggingface_hub
from huggingface_hub import snapshot_download
import sys

folder = snapshot_download("Binamra00/refactoring-forecaster")
sys.path.insert(0, folder)
from forecaster import load

model = load(folder)                 # downloads the two base tokenizers and configs on first use
p = model.probability(code=[skeleton, *methods], history=commit_texts, discourse=discourse_texts)
print(p, p >= model.threshold)

Build the texts the way the dataset does (Notebook 3 of the replication package): the skeleton is the file with every method body replaced by { … } and the package line, imports and comments removed; each method is one text with its Javadoc; a history text is

<COMMIT_MESSAGE>the message</COMMIT_MESSAGE>
<FILE>the file's path in that commit</FILE>
<REMOVED>removed lines</REMOVED>
<ADDED>added lines</ADDED>

leaving out the <REMOVED> or <ADDED> part when it has no lines. History and discourse texts go newest first, and none may be dated at or after the release.

Files

File	Holds
model.safetensors	the trained weights (1.4 GB, float32)
config.json	the windows, the text budget, the threshold and the test scores
forecaster.py	the model classes and load()

Limitations

• Trained on five Java repos (commons-lang, checkstyle, dubbo, junit-framework, questdb); it has not been tested on others.
• The labels come from RefactoringMiner, which misses and misreports some refactorings.
• The probabilities sit above the true share of refactored files, because training used balanced epochs. Compare files by their ranking, or use the threshold above.
• It reads no process or ownership measures, such as how many developers changed a file.

License

The weights are released under Apache-2.0. They are fine-tuned from microsoft/codebert-base (MIT) and sentence-transformers/all-mpnet-base-v2 (Apache-2.0).

Citation

@misc{aryal2026forecaster,
  author = {Aryal, Binamra},
  title  = {Refactoring forecaster (code, history and discourse)},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Binamra00/refactoring-forecaster}}
}
@dataset{aryal2026data,
  author = {Aryal, Binamra},
  title  = {Code, Commit History and Issue/PR Discourse for Forecasting Refactoring},
  year   = {2026},
  version = {2.1},
  publisher = {Zenodo},
  doi    = {10.5281/zenodo.24117444}
}

Acknowledgements

The model fine-tunes CodeBERT (Feng et al., 2020) and all-mpnet-base-v2, a Sentence-BERT model (Reimers and Gurevych, 2019) built on MPNet (Song et al., 2020). Its labels come from RefactoringMiner (Tsantalis et al., 2022). It was trained with Hugging Face Transformers and PyTorch on a RunPod A100.

• Feng, Z. et al. (2020). CodeBERT: A Pre-Trained Model for Programming and Natural Languages. Findings of EMNLP 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.139
• Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP 2019. https://doi.org/10.18653/v1/D19-1410
• Song, K., Tan, X., Qin, T., Lu, J. and Liu, T.-Y. (2020). MPNet: Masked and Permuted Pre-training for Language Understanding. NeurIPS 2020. https://arxiv.org/abs/2004.09297
• Tsantalis, N., Ketkar, A. and Dig, D. (2022). RefactoringMiner 2.0. IEEE Transactions on Software Engineering 48(3). https://doi.org/10.1109/TSE.2020.3007722