Sentence Similarity
sentence-transformers
Safetensors
roberta
feature-extraction
dense
Generated from Trainer
dataset_size:10000
loss:CosineSimilarityLoss
Eval Results (legacy)
text-embeddings-inference
Instructions to use wraithklu/codesentinel-clone-detector with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use wraithklu/codesentinel-clone-detector with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("wraithklu/codesentinel-clone-detector") sentences = [ "private void createTree(DefaultMutableTreeNode top) throws MalformedURLException, ParserConfigurationException, SAXException, IOException {\n InputStream stream;\n URL url = new URL(SHIPS_URL + view.getBaseurl());\n try {\n stream = url.openStream();\n } catch (Exception e) {\n stream = getClass().getResourceAsStream(\"ships.xml\");\n }\n DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();\n DocumentBuilder parser = factory.newDocumentBuilder();\n Document doc = parser.parse(stream);\n NodeList races = doc.getElementsByTagName(\"race\");\n for (int i = 0; i < races.getLength(); i++) {\n Element race = (Element) races.item(i);\n top.add(buildRaceTree(race));\n }\n top.setUserObject(\"Ships\");\n view.getShipTree().repaint();\n view.getShipTree().expandRow(0);\n }", "private GenomicSequence fetch(Chromosome k, int start, int end) throws IOException {\n try {\n String chr = k.toString();\n if (chr.toLowerCase().startsWith(\"chr\")) chr = chr.substring(3);\n SAXParserFactory f = SAXParserFactory.newInstance();\n f.setNamespaceAware(false);\n f.setValidating(false);\n SAXParser parser = f.newSAXParser();\n URL url = new URL(\"http://genome.ucsc.edu/cgi-bin/das/\" + genomeVersion + \"/dna?segment=\" + URLEncoder.encode(chr, \"UTF-8\") + \":\" + (start + 1) + \",\" + (end));\n DASHandler handler = new DASHandler();\n InputStream in = url.openStream();\n parser.parse(in, handler);\n in.close();\n GenomicSequence seq = new GenomicSequence();\n seq.sequence = handler.bytes.toByteArray();\n seq.start = start;\n seq.end = end;\n if (seq.sequence.length != seq.length()) throw new IOException(\"bad bound \" + seq + \" \" + seq.sequence.length + \" \" + seq.length());\n return seq;\n } catch (IOException err) {\n throw err;\n } catch (Exception e) {\n throw new IOException(e);\n }\n }", "public static void main(String[] args) throws Exception {\n FileChannel fc = new FileOutputStream(\"data.txt\").getChannel();\n fc.write(ByteBuffer.wrap(\"some text \".getBytes()));\n fc.close();\n fc = new RandomAccessFile(\"data.txt\", \"rw\").getChannel();\n fc.position(fc.size());\n fc.write(ByteBuffer.wrap(\"some more\".getBytes()));\n fc.close();\n fc = new FileInputStream(\"data.txt\").getChannel();\n ByteBuffer buff = ByteBuffer.allocate(BSIZE);\n fc.read(buff);\n buff.flip();\n while (buff.hasRemaining()) {\n PrintUtil.prt((char) buff.get());\n }\n }", "private FileLog(LOG_LEVEL displayLogLevel, LOG_LEVEL logLevel, String logPath) {\n this.logLevel = logLevel;\n this.displayLogLevel = displayLogLevel;\n if (null != logPath) {\n logFile = new File(logPath, \"current.log\");\n log(LOG_LEVEL.DEBUG, \"FileLog\", \"Initialising logfile \" + logFile.getAbsolutePath() + \" .\");\n try {\n if (logFile.exists()) {\n if (!logFile.renameTo(new File(logPath, System.currentTimeMillis() + \".log\"))) {\n File newFile = new File(logPath, System.currentTimeMillis() + \".log\");\n if (newFile.exists()) {\n log(LOG_LEVEL.WARN, \"FileLog\", \"The file (\" + newFile.getAbsolutePath() + newFile.getName() + \") already exists, will overwrite it.\");\n newFile.delete();\n }\n newFile.createNewFile();\n FileInputStream inStream = new FileInputStream(logFile);\n FileOutputStream outStream = new FileOutputStream(newFile);\n byte buffer[] = null;\n int offSet = 0;\n while (inStream.read(buffer, offSet, 2048) != -1) {\n outStream.write(buffer);\n offSet += 2048;\n }\n inStream.close();\n outStream.close();\n logFile.delete();\n logFile = new File(logPath, \"current.log\");\n }\n }\n logFile.createNewFile();\n } catch (IOException e) {\n logFile = null;\n }\n } else {\n logFile = null;\n }\n }" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
|
Download README.md from wraithklu/codesentinel-clone-detector: direct link, hf CLI and curl.
- Browser
- Download file 48.7 kB
-
https://huggingface.co/wraithklu/codesentinel-clone-detector/resolve/main/README.md
- Command line
-
hf download hf://wraithklu/codesentinel-clone-detector/README.md
-
curl -L -o README.md https://huggingface.co/wraithklu/codesentinel-clone-detector/resolve/main/README.md
48.7 kB
metadata
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- dense
- generated_from_trainer
- dataset_size:10000
- loss:CosineSimilarityLoss
base_model: microsoft/codebert-base
widget:
- source_sentence: >-
private void createTree(DefaultMutableTreeNode top) throws
MalformedURLException, ParserConfigurationException, SAXException,
IOException {
InputStream stream;
URL url = new URL(SHIPS_URL + view.getBaseurl());
try {
stream = url.openStream();
} catch (Exception e) {
stream = getClass().getResourceAsStream("ships.xml");
}
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder parser = factory.newDocumentBuilder();
Document doc = parser.parse(stream);
NodeList races = doc.getElementsByTagName("race");
for (int i = 0; i < races.getLength(); i++) {
Element race = (Element) races.item(i);
top.add(buildRaceTree(race));
}
top.setUserObject("Ships");
view.getShipTree().repaint();
view.getShipTree().expandRow(0);
}
sentences:
- >-
private GenomicSequence fetch(Chromosome k, int start, int end) throws
IOException {
try {
String chr = k.toString();
if (chr.toLowerCase().startsWith("chr")) chr = chr.substring(3);
SAXParserFactory f = SAXParserFactory.newInstance();
f.setNamespaceAware(false);
f.setValidating(false);
SAXParser parser = f.newSAXParser();
URL url = new URL("http://genome.ucsc.edu/cgi-bin/das/" + genomeVersion + "/dna?segment=" + URLEncoder.encode(chr, "UTF-8") + ":" + (start + 1) + "," + (end));
DASHandler handler = new DASHandler();
InputStream in = url.openStream();
parser.parse(in, handler);
in.close();
GenomicSequence seq = new GenomicSequence();
seq.sequence = handler.bytes.toByteArray();
seq.start = start;
seq.end = end;
if (seq.sequence.length != seq.length()) throw new IOException("bad bound " + seq + " " + seq.sequence.length + " " + seq.length());
return seq;
} catch (IOException err) {
throw err;
} catch (Exception e) {
throw new IOException(e);
}
}
- |-
public static void main(String[] args) throws Exception {
FileChannel fc = new FileOutputStream("data.txt").getChannel();
fc.write(ByteBuffer.wrap("some text ".getBytes()));
fc.close();
fc = new RandomAccessFile("data.txt", "rw").getChannel();
fc.position(fc.size());
fc.write(ByteBuffer.wrap("some more".getBytes()));
fc.close();
fc = new FileInputStream("data.txt").getChannel();
ByteBuffer buff = ByteBuffer.allocate(BSIZE);
fc.read(buff);
buff.flip();
while (buff.hasRemaining()) {
PrintUtil.prt((char) buff.get());
}
}
- >-
private FileLog(LOG_LEVEL displayLogLevel, LOG_LEVEL logLevel, String
logPath) {
this.logLevel = logLevel;
this.displayLogLevel = displayLogLevel;
if (null != logPath) {
logFile = new File(logPath, "current.log");
log(LOG_LEVEL.DEBUG, "FileLog", "Initialising logfile " + logFile.getAbsolutePath() + " .");
try {
if (logFile.exists()) {
if (!logFile.renameTo(new File(logPath, System.currentTimeMillis() + ".log"))) {
File newFile = new File(logPath, System.currentTimeMillis() + ".log");
if (newFile.exists()) {
log(LOG_LEVEL.WARN, "FileLog", "The file (" + newFile.getAbsolutePath() + newFile.getName() + ") already exists, will overwrite it.");
newFile.delete();
}
newFile.createNewFile();
FileInputStream inStream = new FileInputStream(logFile);
FileOutputStream outStream = new FileOutputStream(newFile);
byte buffer[] = null;
int offSet = 0;
while (inStream.read(buffer, offSet, 2048) != -1) {
outStream.write(buffer);
offSet += 2048;
}
inStream.close();
outStream.close();
logFile.delete();
logFile = new File(logPath, "current.log");
}
}
logFile.createNewFile();
} catch (IOException e) {
logFile = null;
}
} else {
logFile = null;
}
}
- source_sentence: |-
private ByteBuffer readProgram(URL url) throws IOException {
StringBuilder program = new StringBuilder();
BufferedReader in = null;
try {
in = new BufferedReader(new InputStreamReader(url.openStream()));
String line;
while ((line = in.readLine()) != null) {
program.append(line).append("\n");
}
} finally {
if (in != null) in.close();
}
ByteBuffer buffer = BufferUtils.createByteBuffer(program.length());
for (int i = 0; i < program.length(); i++) {
buffer.put((byte) (program.charAt(i) & 0xFF));
}
buffer.flip();
return buffer;
}
sentences:
- |-
public int delete(BusinessObject o) throws DAOException {
int delete = 0;
Currency curr = (Currency) o;
try {
PreparedStatement pst = connection.prepareStatement(XMLGetQuery.getQuery("DELETE_CURRENCY"));
pst.setInt(1, curr.getId());
delete = pst.executeUpdate();
if (delete <= 0) {
connection.rollback();
throw new DAOException("Number of rows <= 0");
} else if (delete > 1) {
connection.rollback();
throw new DAOException("Number of rows > 1");
}
connection.commit();
} catch (SQLException e) {
Log.write(e.getMessage());
throw new DAOException("A SQLException has occured");
} catch (NullPointerException npe) {
Log.write(npe.getMessage());
throw new DAOException("Connection null");
}
return delete;
}
- >-
public static String encrypt(String plaintext) throws
EncryptionException {
if (plaintext == null) {
throw new EncryptionException();
}
try {
final MessageDigest md = MessageDigest.getInstance("SHA");
md.update(plaintext.getBytes("UTF-8"));
return Base64.encodeBytes(md.digest());
} catch (NoSuchAlgorithmException e) {
throw new EncryptionException(e);
} catch (UnsupportedEncodingException e) {
throw new EncryptionException(e);
}
}
- |-
public static void copyFile(File in, File out) throws IOException {
FileChannel sourceChannel = new FileInputStream(in).getChannel();
FileChannel destinationChannel = new FileOutputStream(out).getChannel();
destinationChannel.transferFrom(sourceChannel, 0, sourceChannel.size());
sourceChannel.close();
destinationChannel.close();
}
- source_sentence: >-
private void sortWhats(String[] labels, int[] whats, String
simplifyString) {
int n = whats.length;
boolean swapped;
do {
swapped = false;
for (int i = 0; i < n - 1; i++) {
int i0_pos = simplifyString.indexOf(labels[whats[i]]);
int i1_pos = simplifyString.indexOf(labels[whats[i + 1]]);
if (i0_pos > i1_pos) {
int temp = whats[i];
whats[i] = whats[i + 1];
whats[i + 1] = temp;
swapped = true;
}
}
} while (swapped);
}
sentences:
- |-
public String GetMemberName(String id) {
String name = null;
try {
String line;
URL url = new URL(intvasmemberDeatails + "?CID=" + id);
URLConnection connection = url.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while ((line = reader.readLine()) != null) {
name = line;
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
String[] parts = name.split(" ");
rating = parts[2];
return parts[0] + " " + parts[1];
}
- >-
private static void readAndRewrite(File inFile, File outFile) throws
IOException {
ImageInputStream iis = ImageIO.createImageInputStream(new BufferedInputStream(new FileInputStream(inFile)));
DcmParser dcmParser = DcmParserFactory.getInstance().newDcmParser(iis);
Dataset ds = DcmObjectFactory.getInstance().newDataset();
dcmParser.setDcmHandler(ds.getDcmHandler());
dcmParser.parseDcmFile(null, Tags.PixelData);
PixelDataReader pdReader = pdFact.newReader(ds, iis, dcmParser.getDcmDecodeParam().byteOrder, dcmParser.getReadVR());
System.out.println("reading " + inFile + "...");
pdReader.readPixelData(false);
ImageOutputStream out = ImageIO.createImageOutputStream(new BufferedOutputStream(new FileOutputStream(outFile)));
DcmEncodeParam dcmEncParam = DcmEncodeParam.IVR_LE;
ds.writeDataset(out, dcmEncParam);
ds.writeHeader(out, dcmEncParam, Tags.PixelData, dcmParser.getReadVR(), dcmParser.getReadLength());
System.out.println("writing " + outFile + "...");
PixelDataWriter pdWriter = pdFact.newWriter(pdReader.getPixelDataArray(), false, ds, out, dcmParser.getDcmDecodeParam().byteOrder, dcmParser.getReadVR());
pdWriter.writePixelData();
out.flush();
out.close();
System.out.println("done!");
}
- |-
public void insertComponent() throws SQLException {
Connection connection = null;
PreparedStatement ps = null;
try {
Class.forName("com.mysql.jdbc.Driver");
connection = (Connection) DriverManager.getConnection(this.jdbcURL);
connection.setAutoCommit(false);
String query = "INSERT INTO components(name,rate,quantity, description) VALUES(?,?,?,?)";
ps = (PreparedStatement) connection.prepareStatement(query);
ps.setString(1, this.name);
ps.setDouble(2, this.rate);
ps.setInt(3, this.quantity);
ps.setString(4, this.description);
ps.executeUpdate();
connection.commit();
} catch (Exception ex) {
connection.rollback();
} finally {
try {
connection.close();
} catch (Exception ex) {
}
try {
ps.close();
} catch (Exception ex) {
}
}
}
- source_sentence: |-
public static String toMD5(String seed) {
MessageDigest md5 = null;
StringBuffer temp_sb = null;
try {
md5 = MessageDigest.getInstance("MD5");
md5.update(seed.getBytes());
byte[] array = md5.digest();
temp_sb = new StringBuffer();
for (int i = 0; i < array.length; i++) {
int b = array[i] & 0xFF;
if (b < 0x10) temp_sb.append('0');
temp_sb.append(Integer.toHexString(b));
}
} catch (NoSuchAlgorithmException err) {
err.printStackTrace();
}
return temp_sb.toString();
}
sentences:
- >-
public static boolean checkEncryptedPassword(String md5key, String
passwordAccount, String encryptedPassword, int passwdenc) throws
NoSuchAlgorithmException, UnsupportedEncodingException {
MessageDigest md = MessageDigest.getInstance(Constants.ALGORITHM);
switch(passwdenc) {
case 1:
md.update((md5key + encryptedPassword).getBytes("8859_1"));
return md.digest().equals(passwordAccount.getBytes("8859_1"));
case 2:
md.update((encryptedPassword + md5key).getBytes("8859_1"));
return md.digest().equals(passwordAccount.getBytes("8859_1"));
default:
return false;
}
}
- |-
@Test
public void shouldDownloadFileUsingPublicLink() throws Exception {
String bucketName = "test-" + UUID.randomUUID();
Service service = new WebClientService(credentials);
service.createBucket(bucketName);
File file = folder.newFile("foo.txt");
FileUtils.writeStringToFile(file, UUID.randomUUID().toString());
service.createObject(bucketName, file.getName(), file, new NullProgressListener());
String publicUrl = service.getPublicUrl(bucketName, file.getName(), new DateTime().plusDays(5));
File saved = folder.newFile("saved.txt");
InputStream input = new URL(publicUrl).openConnection().getInputStream();
FileOutputStream output = new FileOutputStream(saved);
IOUtils.copy(input, output);
output.close();
assertThat("Corrupted download", Files.computeMD5(saved), equalTo(Files.computeMD5(file)));
service.deleteObject(bucketName, file.getName());
service.deleteBucket(bucketName);
}
- |-
public static void main(String[] args) {
try {
boolean readExp = Utils.getFlag('l', args);
final boolean writeExp = Utils.getFlag('s', args);
final String expFile = Utils.getOption('f', args);
if ((readExp || writeExp) && (expFile.length() == 0)) {
throw new Exception("A filename must be given with the -f option");
}
Experiment exp = null;
if (readExp) {
FileInputStream fi = new FileInputStream(expFile);
ObjectInputStream oi = new ObjectInputStream(new BufferedInputStream(fi));
exp = (Experiment) oi.readObject();
oi.close();
} else {
exp = new Experiment();
}
System.err.println("Initial Experiment:\n" + exp.toString());
final JFrame jf = new JFrame("Weka Experiment Setup");
jf.getContentPane().setLayout(new BorderLayout());
final SetupPanel sp = new SetupPanel();
jf.getContentPane().add(sp, BorderLayout.CENTER);
jf.addWindowListener(new WindowAdapter() {
public void windowClosing(WindowEvent e) {
System.err.println("\nFinal Experiment:\n" + sp.m_Exp.toString());
if (writeExp) {
try {
FileOutputStream fo = new FileOutputStream(expFile);
ObjectOutputStream oo = new ObjectOutputStream(new BufferedOutputStream(fo));
oo.writeObject(sp.m_Exp);
oo.close();
} catch (Exception ex) {
ex.printStackTrace();
System.err.println("Couldn't write experiment to: " + expFile + '\n' + ex.getMessage());
}
}
jf.dispose();
System.exit(0);
}
});
jf.pack();
jf.setVisible(true);
System.err.println("Short nap");
Thread.currentThread().sleep(3000);
System.err.println("Done");
sp.setExperiment(exp);
} catch (Exception ex) {
ex.printStackTrace();
System.err.println(ex.getMessage());
}
}
- source_sentence: >-
protected void serveStaticContent(HttpServletRequest request,
HttpServletResponse response, String pathInfo) throws ServletException {
InputStream is = servletConfig.getServletContext().getResourceAsStream(pathInfo);
if (is == null) {
throw new ServletException("Static resource " + pathInfo + " is not available");
}
try {
int ind = pathInfo.lastIndexOf(".");
if (ind != -1 && ind < pathInfo.length()) {
String type = STATIC_CONTENT_TYPES.get(pathInfo.substring(ind + 1));
if (type != null) {
response.setContentType(type);
}
}
ServletOutputStream os = response.getOutputStream();
IOUtils.copy(is, os);
os.flush();
} catch (IOException ex) {
throw new ServletException("Static resource " + pathInfo + " can not be written to the output stream");
}
}
sentences:
- >-
private static String makeLoginRequest(String authenticationUrl, String
accountEmail, String accountPassword) throws IOException {
URL url = new URL(authenticationUrl);
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
urlConnection.setUseCaches(false);
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
StringBuffer content = new StringBuffer();
content.append("Email=").append(URLEncoder.encode(accountEmail, "UTF-8"));
content.append("&Passwd=").append(URLEncoder.encode(accountPassword, "UTF-8"));
content.append("&source=").append(URLEncoder.encode("Google Base data API for OFBiz", "UTF-8"));
content.append("&service=").append(URLEncoder.encode("gbase", "UTF-8"));
OutputStream outputStream = urlConnection.getOutputStream();
outputStream.write(content.toString().getBytes("UTF-8"));
outputStream.close();
int responseCode = urlConnection.getResponseCode();
InputStream inputStream;
if (responseCode == HttpURLConnection.HTTP_OK) {
inputStream = urlConnection.getInputStream();
} else {
inputStream = urlConnection.getErrorStream();
}
return toString(inputStream);
}
- |-
public void constructFundamentalView() {
String className;
String methodName;
String field;
boolean foundRead = false;
boolean foundWrite = false;
boolean classWritten = false;
try {
FundView = new BufferedWriter(new FileWriter("InfoFiles/FundamentalView.txt"));
FileInputStream fstreamPC = new FileInputStream("InfoFiles/PrincipleClassGroup.txt");
DataInputStream inPC = new DataInputStream(fstreamPC);
BufferedReader PC = new BufferedReader(new InputStreamReader(inPC));
while ((field = PC.readLine()) != null) {
className = field;
FundView.write(className);
FundView.newLine();
classWritten = true;
while ((methodName = PC.readLine()) != null) {
if (methodName.contentEquals("EndOfClass")) break;
FundView.write("StartOfMethod");
FundView.newLine();
FundView.write(methodName);
FundView.newLine();
for (int i = 0; i < readFileCount && foundRead == false; i++) {
if (methodName.compareTo(readArray[i]) == 0) {
foundRead = true;
for (int j = 1; readArray[i + j].compareTo("EndOfMethod") != 0; j++) {
if (readArray[i + j].indexOf(".") > 0) {
field = readArray[i + j].substring(0, readArray[i + j].indexOf("."));
if (field.compareTo(className) == 0) {
FundView.write(readArray[i + j]);
FundView.newLine();
}
}
}
}
}
for (int i = 0; i < writeFileCount && foundWrite == false; i++) {
if (methodName.compareTo(writeArray[i]) == 0) {
foundWrite = true;
for (int j = 1; writeArray[i + j].compareTo("EndOfMethod") != 0; j++) {
if (writeArray[i + j].indexOf(".") > 0) {
field = writeArray[i + j].substring(0, writeArray[i + j].indexOf("."));
if (field.compareTo(className) == 0) {
FundView.write(writeArray[i + j]);
FundView.newLine();
}
}
}
}
}
FundView.write("EndOfMethod");
FundView.newLine();
foundRead = false;
foundWrite = false;
}
if (classWritten == true) {
FundView.write("EndOfClass");
FundView.newLine();
classWritten = false;
}
}
PC.close();
FundView.close();
} catch (IOException e) {
e.printStackTrace();
}
}
- |-
public static void main(String[] args) throws Exception {
if (args.length != 2) {
System.out.println("arguments: sourcefile destfile");
System.exit(1);
}
FileChannel in = new FileInputStream(args[0]).getChannel(), out = new FileOutputStream(args[1]).getChannel();
in.transferTo(0, in.size(), out);
}
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- pearson_cosine
- spearman_cosine
model-index:
- name: SentenceTransformer based on microsoft/codebert-base
results:
- task:
type: semantic-similarity
name: Semantic Similarity
dataset:
name: bigclonebench validation
type: bigclonebench-validation
metrics:
- type: pearson_cosine
value: 0.9050117464281175
name: Pearson Cosine
- type: spearman_cosine
value: 0.8342891483192489
name: Spearman Cosine
SentenceTransformer based on microsoft/codebert-base
This is a sentence-transformers model finetuned from microsoft/codebert-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: microsoft/codebert-base
- Maximum Sequence Length: 256 tokens
- Output Dimensionality: 768 dimensions
- Similarity Function: Cosine Similarity
- Supported Modality: Text
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RobertaModel'})
(1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("wraithklu/codesentinel-clone-detector")
# Run inference
sentences = [
'protected void serveStaticContent(HttpServletRequest request, HttpServletResponse response, String pathInfo) throws ServletException {\n InputStream is = servletConfig.getServletContext().getResourceAsStream(pathInfo);\n if (is == null) {\n throw new ServletException("Static resource " + pathInfo + " is not available");\n }\n try {\n int ind = pathInfo.lastIndexOf(".");\n if (ind != -1 && ind < pathInfo.length()) {\n String type = STATIC_CONTENT_TYPES.get(pathInfo.substring(ind + 1));\n if (type != null) {\n response.setContentType(type);\n }\n }\n ServletOutputStream os = response.getOutputStream();\n IOUtils.copy(is, os);\n os.flush();\n } catch (IOException ex) {\n throw new ServletException("Static resource " + pathInfo + " can not be written to the output stream");\n }\n }',
'private static String makeLoginRequest(String authenticationUrl, String accountEmail, String accountPassword) throws IOException {\n URL url = new URL(authenticationUrl);\n HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();\n urlConnection.setRequestMethod("POST");\n urlConnection.setDoInput(true);\n urlConnection.setDoOutput(true);\n urlConnection.setUseCaches(false);\n urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");\n StringBuffer content = new StringBuffer();\n content.append("Email=").append(URLEncoder.encode(accountEmail, "UTF-8"));\n content.append("&Passwd=").append(URLEncoder.encode(accountPassword, "UTF-8"));\n content.append("&source=").append(URLEncoder.encode("Google Base data API for OFBiz", "UTF-8"));\n content.append("&service=").append(URLEncoder.encode("gbase", "UTF-8"));\n OutputStream outputStream = urlConnection.getOutputStream();\n outputStream.write(content.toString().getBytes("UTF-8"));\n outputStream.close();\n int responseCode = urlConnection.getResponseCode();\n InputStream inputStream;\n if (responseCode == HttpURLConnection.HTTP_OK) {\n inputStream = urlConnection.getInputStream();\n } else {\n inputStream = urlConnection.getErrorStream();\n }\n return toString(inputStream);\n }',
'public static void main(String[] args) throws Exception {\n if (args.length != 2) {\n System.out.println("arguments: sourcefile destfile");\n System.exit(1);\n }\n FileChannel in = new FileInputStream(args[0]).getChannel(), out = new FileOutputStream(args[1]).getChannel();\n in.transferTo(0, in.size(), out);\n }',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000, -0.0114, 0.9605],
# [-0.0114, 1.0000, 0.0068],
# [ 0.9605, 0.0068, 1.0000]])
Evaluation
Metrics
Semantic Similarity
- Dataset:
bigclonebench-validation - Evaluated with
EmbeddingSimilarityEvaluator
| Metric | Value |
|---|---|
| pearson_cosine | 0.905 |
| spearman_cosine | 0.8343 |
Training Details
Training Dataset
Unnamed Dataset
- Size: 10,000 training samples
- Columns:
sentence_0,sentence_1, andlabel - Approximate statistics based on the first 100 samples:
sentence_0 sentence_1 label type string string float modality text text details - min: 134 tokens
- mean: 240.02 tokens
- max: 256 tokens
- min: 104 tokens
- mean: 237.68 tokens
- max: 256 tokens
- min: 0.0
- mean: 0.49
- max: 1.0
- Samples:
sentence_0 sentence_1 label public InputLattice(String file, int type, boolean convertJoinMeet) throws FileNotFoundException, IOException {
InputStream latStream = null;
labels = new Vector();
upperCoversVector = new Vector();
Vector upperCovers = new Vector();
edgeColors = new Hashtable();
Vector upperCovering = new Vector();
int level = 0;
String str;
String currentElt = null;
String cover = null;
boolean first = true;
StreamTokenizer in;
switch(type) {
case FILE:
latStream = new FileInputStream(file);
break;
case STRING:
latStream = new StringBufferInputStream(file);
break;
case URL:
URL url = new URL("http", Config.HOST, Config.PORT, "/" + Config.HASSE + "/" + Config.URL_STREAMER + file);
System.out.println("url is " + url);
URLConnection connection = url.openCo...private String calculateHash(String s) {
if (s == null) {
return null;
}
MessageDigest messageDigest;
try {
messageDigest = MessageDigest.getInstance("SHA");
} catch (NoSuchAlgorithmException e) {
logger.error("Could not find a message digest algorithm.");
return null;
}
messageDigest.update(s.getBytes());
byte[] hash = messageDigest.digest();
StringBuilder sb = new StringBuilder();
for (byte b : hash) {
sb.append(String.format("%02x", b));
}
return sb.toString();
}0.0void load(URL url) throws IOException {
BufferedReader r = new BufferedReader(new InputStreamReader(url.openStream()));
Vector3f scale = new Vector3f(1, 1, 1);
Group currentGroup = new Group();
currentGroup.name = "default";
groups.add(currentGroup);
String line;
while ((line = r.readLine()) != null) {
String[] params = line.split(" +");
if (params.length == 0) continue;
String command = params[0];
if (params[0].equals("v")) {
Vector3f vertex = new Vector3f(Float.parseFloat(params[1]) * scale.x, Float.parseFloat(params[2]) * scale.y, Float.parseFloat(params[3]) * scale.z);
verticies.add(vertex);
radius = Math.max(radius, vertex.length());
}
if (command.equals("center")) {
epicenter = new Vector3f(Float.parseFloat(params[1]), Float.parseFloat(params[2]), Float.parseFloat(params[3]));
} ...public static boolean canWeConnectToInternet() {
String s = "http://www.google.com/";
URL url = null;
boolean can = false;
URLConnection conection = null;
try {
url = new URL(s);
} catch (MalformedURLException e) {
System.out.println("This should never happend. Error in URL name. URL specified was:" + s + ".");
}
try {
conection = url.openConnection();
conection.connect();
can = true;
} catch (IOException e) {
can = false;
}
if (can) {
}
return can;
}0.0public static java.io.ByteArrayOutputStream getFileByteStream(URL _url) {
java.io.ByteArrayOutputStream buffer = new java.io.ByteArrayOutputStream();
try {
InputStream input = _url.openStream();
IOUtils.copy(input, buffer);
IOUtils.closeQuietly(input);
} catch (Exception err) {
throw new RuntimeException(err);
}
return buffer;
}public BufferedImage processUsingTemp(InputStream input, DjatokaDecodeParam params) throws DjatokaException {
File in;
try {
in = File.createTempFile("tmp", ".jp2");
FileOutputStream fos = new FileOutputStream(in);
in.deleteOnExit();
IOUtils.copyStream(input, fos);
} catch (IOException e) {
logger.error(e, e);
throw new DjatokaException(e);
}
BufferedImage bi = process(in.getAbsolutePath(), params);
if (in != null) in.delete();
return bi;
}1.0 - Loss:
CosineSimilarityLosswith these parameters:{ "loss_fct": "torch.nn.modules.loss.MSELoss", "cos_score_transformation": "torch.nn.modules.linear.Identity" }
Training Hyperparameters
Non-Default Hyperparameters
per_device_train_batch_size: 32num_train_epochs: 2fp16: Trueper_device_eval_batch_size: 32multi_dataset_batch_sampler: round_robin
All Hyperparameters
Click to expand
per_device_train_batch_size: 32num_train_epochs: 2max_steps: -1learning_rate: 5e-05lr_scheduler_type: linearlr_scheduler_kwargs: Nonewarmup_steps: 0optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 1average_tokens_across_devices: Truemax_grad_norm: 1label_smoothing_factor: 0.0bf16: Falsefp16: Truebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: Nonetrackio_bucket_id: Nonetrackio_static_space_id: Noneper_device_eval_batch_size: 32prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: Noneuse_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Falsedataloader_num_workers: 0dataloader_pin_memory: Truedataloader_persistent_workers: Falsedataloader_prefetch_factor: Nonedataloader_multiprocessing_context: Nonedataloader_in_order: Trueremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_static_graph: Noneddp_backend: Noneddp_timeout: 1800fsdp: Nonefsdp_config: Nonedeepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonelocal_rank: -1prompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: round_robinrouter_mapping: {}learning_rate_mapping: {}warmup_ratio: None
Training Logs
| Epoch | Step | Training Loss | bigclonebench-validation_spearman_cosine |
|---|---|---|---|
| 0.4984 | 156 | - | 0.8142 |
| 0.9968 | 312 | - | 0.8325 |
| 1.0 | 313 | - | 0.8325 |
| 1.4952 | 468 | - | 0.8346 |
| 1.5974 | 500 | 0.0901 | - |
| 1.9936 | 624 | - | 0.8343 |
| 2.0 | 626 | - | 0.8343 |
Training Time
- Training: 13.3 minutes
Framework Versions
- Python: 3.13.16
- Sentence Transformers: 5.7.0
- Transformers: 5.22.1
- PyTorch: 2.14.1+cu130
- Accelerate: 1.15.0
- Datasets: 4.8.5
- Tokenizers: 0.23.3
Additional Resources
- Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
- Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
- Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
- Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
- Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}