CodeX-SFT

A code similarity model that measures whether two programs solve a problem with the same algorithmic approach. It is the metric behind the diversity numbers in DivAlgo: A systematic evaluation of algorithmic diversity for code generation models (arXiv:2610.19157); code and evaluation scripts are at https://github.com/anubhav-jangra/divalgo.

Usage

import json, torch, torch.nn.functional as F
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, AutoModel

name = "jangraanubhav/codex-sft"
tok = AutoTokenizer.from_pretrained(name)
enc = AutoModel.from_pretrained(name, trust_remote_code=True, torch_dtype=torch.bfloat16).eval()
cal = json.load(open(hf_hub_download(name, "calibration.json")))

def embed(codes):
    x = tok(codes, truncation=True, max_length=2048, padding=True, return_tensors="pt")
    with torch.no_grad():
        h = enc(**x).last_hidden_state
    m = x["attention_mask"].unsqueeze(-1).to(h.dtype)
    return F.normalize((h * m).sum(1) / m.sum(1), dim=-1).float()

e = embed([code_a, code_b])
cos = float(e[0] @ e[1])
distance = cal["softplus_a"] * (1 - cos) / 2 + cal["b"]          # calibrated distance in [0, 1]
distinct = distance >= cal["threshold_distance"]                  # 0.3856

Ranking by distance is the same as ranking by cosine, so the embeddings can also be used directly, for example as the kernel of a Vendi score over a pool of solutions.

Citation

@article{jangra2026divalgo,
  title   = {DivAlgo: A systematic evaluation of algorithmic diversity for code generation models},
  author  = {Jangra, Anubhav and Muresan, Smaranda},
  journal = {arXiv preprint arXiv:2610.19157},
  year    = {2026}
}