Sentence Similarity
sentence-transformers
Safetensors
roberta
feature-extraction
dense
Generated from Trainer
dataset_size:10000
loss:CosineSimilarityLoss
Eval Results (legacy)
text-embeddings-inference
Instructions to use wraithklu/codesentinel-clone-detector with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use wraithklu/codesentinel-clone-detector with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("wraithklu/codesentinel-clone-detector") sentences = [ "private void createTree(DefaultMutableTreeNode top) throws MalformedURLException, ParserConfigurationException, SAXException, IOException {\n InputStream stream;\n URL url = new URL(SHIPS_URL + view.getBaseurl());\n try {\n stream = url.openStream();\n } catch (Exception e) {\n stream = getClass().getResourceAsStream(\"ships.xml\");\n }\n DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();\n DocumentBuilder parser = factory.newDocumentBuilder();\n Document doc = parser.parse(stream);\n NodeList races = doc.getElementsByTagName(\"race\");\n for (int i = 0; i < races.getLength(); i++) {\n Element race = (Element) races.item(i);\n top.add(buildRaceTree(race));\n }\n top.setUserObject(\"Ships\");\n view.getShipTree().repaint();\n view.getShipTree().expandRow(0);\n }", "private GenomicSequence fetch(Chromosome k, int start, int end) throws IOException {\n try {\n String chr = k.toString();\n if (chr.toLowerCase().startsWith(\"chr\")) chr = chr.substring(3);\n SAXParserFactory f = SAXParserFactory.newInstance();\n f.setNamespaceAware(false);\n f.setValidating(false);\n SAXParser parser = f.newSAXParser();\n URL url = new URL(\"http://genome.ucsc.edu/cgi-bin/das/\" + genomeVersion + \"/dna?segment=\" + URLEncoder.encode(chr, \"UTF-8\") + \":\" + (start + 1) + \",\" + (end));\n DASHandler handler = new DASHandler();\n InputStream in = url.openStream();\n parser.parse(in, handler);\n in.close();\n GenomicSequence seq = new GenomicSequence();\n seq.sequence = handler.bytes.toByteArray();\n seq.start = start;\n seq.end = end;\n if (seq.sequence.length != seq.length()) throw new IOException(\"bad bound \" + seq + \" \" + seq.sequence.length + \" \" + seq.length());\n return seq;\n } catch (IOException err) {\n throw err;\n } catch (Exception e) {\n throw new IOException(e);\n }\n }", "public static void main(String[] args) throws Exception {\n FileChannel fc = new FileOutputStream(\"data.txt\").getChannel();\n fc.write(ByteBuffer.wrap(\"some text \".getBytes()));\n fc.close();\n fc = new RandomAccessFile(\"data.txt\", \"rw\").getChannel();\n fc.position(fc.size());\n fc.write(ByteBuffer.wrap(\"some more\".getBytes()));\n fc.close();\n fc = new FileInputStream(\"data.txt\").getChannel();\n ByteBuffer buff = ByteBuffer.allocate(BSIZE);\n fc.read(buff);\n buff.flip();\n while (buff.hasRemaining()) {\n PrintUtil.prt((char) buff.get());\n }\n }", "private FileLog(LOG_LEVEL displayLogLevel, LOG_LEVEL logLevel, String logPath) {\n this.logLevel = logLevel;\n this.displayLogLevel = displayLogLevel;\n if (null != logPath) {\n logFile = new File(logPath, \"current.log\");\n log(LOG_LEVEL.DEBUG, \"FileLog\", \"Initialising logfile \" + logFile.getAbsolutePath() + \" .\");\n try {\n if (logFile.exists()) {\n if (!logFile.renameTo(new File(logPath, System.currentTimeMillis() + \".log\"))) {\n File newFile = new File(logPath, System.currentTimeMillis() + \".log\");\n if (newFile.exists()) {\n log(LOG_LEVEL.WARN, \"FileLog\", \"The file (\" + newFile.getAbsolutePath() + newFile.getName() + \") already exists, will overwrite it.\");\n newFile.delete();\n }\n newFile.createNewFile();\n FileInputStream inStream = new FileInputStream(logFile);\n FileOutputStream outStream = new FileOutputStream(newFile);\n byte buffer[] = null;\n int offSet = 0;\n while (inStream.read(buffer, offSet, 2048) != -1) {\n outStream.write(buffer);\n offSet += 2048;\n }\n inStream.close();\n outStream.close();\n logFile.delete();\n logFile = new File(logPath, \"current.log\");\n }\n }\n logFile.createNewFile();\n } catch (IOException e) {\n logFile = null;\n }\n } else {\n logFile = null;\n }\n }" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
|
Download README.md from wraithklu/codesentinel-clone-detector: direct link, hf CLI and curl.
- Browser
- Download file 48.7 kB
-
https://huggingface.co/wraithklu/codesentinel-clone-detector/resolve/main/README.md
- Command line
-
hf download hf://wraithklu/codesentinel-clone-detector/README.md
-
curl -L -o README.md https://huggingface.co/wraithklu/codesentinel-clone-detector/resolve/main/README.md
48.7 kB
| tags: | |
| - sentence-transformers | |
| - sentence-similarity | |
| - feature-extraction | |
| - dense | |
| - generated_from_trainer | |
| - dataset_size:10000 | |
| - loss:CosineSimilarityLoss | |
| base_model: microsoft/codebert-base | |
| widget: | |
| - source_sentence: "private void createTree(DefaultMutableTreeNode top) throws MalformedURLException,\ | |
| \ ParserConfigurationException, SAXException, IOException {\n InputStream\ | |
| \ stream;\n URL url = new URL(SHIPS_URL + view.getBaseurl());\n \ | |
| \ try {\n stream = url.openStream();\n } catch (Exception e)\ | |
| \ {\n stream = getClass().getResourceAsStream(\"ships.xml\");\n \ | |
| \ }\n DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();\n\ | |
| \ DocumentBuilder parser = factory.newDocumentBuilder();\n Document\ | |
| \ doc = parser.parse(stream);\n NodeList races = doc.getElementsByTagName(\"\ | |
| race\");\n for (int i = 0; i < races.getLength(); i++) {\n Element\ | |
| \ race = (Element) races.item(i);\n top.add(buildRaceTree(race));\n\ | |
| \ }\n top.setUserObject(\"Ships\");\n view.getShipTree().repaint();\n\ | |
| \ view.getShipTree().expandRow(0);\n }" | |
| sentences: | |
| - "private GenomicSequence fetch(Chromosome k, int start, int end) throws IOException\ | |
| \ {\n try {\n String chr = k.toString();\n if (chr.toLowerCase().startsWith(\"\ | |
| chr\")) chr = chr.substring(3);\n SAXParserFactory f = SAXParserFactory.newInstance();\n\ | |
| \ f.setNamespaceAware(false);\n f.setValidating(false);\n\ | |
| \ SAXParser parser = f.newSAXParser();\n URL url = new URL(\"\ | |
| http://genome.ucsc.edu/cgi-bin/das/\" + genomeVersion + \"/dna?segment=\" + URLEncoder.encode(chr,\ | |
| \ \"UTF-8\") + \":\" + (start + 1) + \",\" + (end));\n DASHandler handler\ | |
| \ = new DASHandler();\n InputStream in = url.openStream();\n \ | |
| \ parser.parse(in, handler);\n in.close();\n GenomicSequence\ | |
| \ seq = new GenomicSequence();\n seq.sequence = handler.bytes.toByteArray();\n\ | |
| \ seq.start = start;\n seq.end = end;\n if (seq.sequence.length\ | |
| \ != seq.length()) throw new IOException(\"bad bound \" + seq + \" \" + seq.sequence.length\ | |
| \ + \" \" + seq.length());\n return seq;\n } catch (IOException\ | |
| \ err) {\n throw err;\n } catch (Exception e) {\n \ | |
| \ throw new IOException(e);\n }\n }" | |
| - "public static void main(String[] args) throws Exception {\n FileChannel\ | |
| \ fc = new FileOutputStream(\"data.txt\").getChannel();\n fc.write(ByteBuffer.wrap(\"\ | |
| some text \".getBytes()));\n fc.close();\n fc = new RandomAccessFile(\"\ | |
| data.txt\", \"rw\").getChannel();\n fc.position(fc.size());\n fc.write(ByteBuffer.wrap(\"\ | |
| some more\".getBytes()));\n fc.close();\n fc = new FileInputStream(\"\ | |
| data.txt\").getChannel();\n ByteBuffer buff = ByteBuffer.allocate(BSIZE);\n\ | |
| \ fc.read(buff);\n buff.flip();\n while (buff.hasRemaining())\ | |
| \ {\n PrintUtil.prt((char) buff.get());\n }\n }" | |
| - "private FileLog(LOG_LEVEL displayLogLevel, LOG_LEVEL logLevel, String logPath)\ | |
| \ {\n this.logLevel = logLevel;\n this.displayLogLevel = displayLogLevel;\n\ | |
| \ if (null != logPath) {\n logFile = new File(logPath, \"current.log\"\ | |
| );\n log(LOG_LEVEL.DEBUG, \"FileLog\", \"Initialising logfile \" +\ | |
| \ logFile.getAbsolutePath() + \" .\");\n try {\n if\ | |
| \ (logFile.exists()) {\n if (!logFile.renameTo(new File(logPath,\ | |
| \ System.currentTimeMillis() + \".log\"))) {\n File newFile\ | |
| \ = new File(logPath, System.currentTimeMillis() + \".log\");\n \ | |
| \ if (newFile.exists()) {\n log(LOG_LEVEL.WARN,\ | |
| \ \"FileLog\", \"The file (\" + newFile.getAbsolutePath() + newFile.getName()\ | |
| \ + \") already exists, will overwrite it.\");\n newFile.delete();\n\ | |
| \ }\n newFile.createNewFile();\n\ | |
| \ FileInputStream inStream = new FileInputStream(logFile);\n\ | |
| \ FileOutputStream outStream = new FileOutputStream(newFile);\n\ | |
| \ byte buffer[] = null;\n int offSet\ | |
| \ = 0;\n while (inStream.read(buffer, offSet, 2048) !=\ | |
| \ -1) {\n outStream.write(buffer);\n \ | |
| \ offSet += 2048;\n }\n \ | |
| \ inStream.close();\n outStream.close();\n \ | |
| \ logFile.delete();\n logFile = new File(logPath,\ | |
| \ \"current.log\");\n }\n }\n \ | |
| \ logFile.createNewFile();\n } catch (IOException e) {\n \ | |
| \ logFile = null;\n }\n } else {\n logFile\ | |
| \ = null;\n }\n }" | |
| - source_sentence: "private ByteBuffer readProgram(URL url) throws IOException {\n\ | |
| \ StringBuilder program = new StringBuilder();\n BufferedReader\ | |
| \ in = null;\n try {\n in = new BufferedReader(new InputStreamReader(url.openStream()));\n\ | |
| \ String line;\n while ((line = in.readLine()) != null)\ | |
| \ {\n program.append(line).append(\"\\n\");\n }\n \ | |
| \ } finally {\n if (in != null) in.close();\n }\n \ | |
| \ ByteBuffer buffer = BufferUtils.createByteBuffer(program.length());\n \ | |
| \ for (int i = 0; i < program.length(); i++) {\n buffer.put((byte)\ | |
| \ (program.charAt(i) & 0xFF));\n }\n buffer.flip();\n return\ | |
| \ buffer;\n }" | |
| sentences: | |
| - "public int delete(BusinessObject o) throws DAOException {\n int delete\ | |
| \ = 0;\n Currency curr = (Currency) o;\n try {\n PreparedStatement\ | |
| \ pst = connection.prepareStatement(XMLGetQuery.getQuery(\"DELETE_CURRENCY\"));\n\ | |
| \ pst.setInt(1, curr.getId());\n delete = pst.executeUpdate();\n\ | |
| \ if (delete <= 0) {\n connection.rollback();\n \ | |
| \ throw new DAOException(\"Number of rows <= 0\");\n } else\ | |
| \ if (delete > 1) {\n connection.rollback();\n throw\ | |
| \ new DAOException(\"Number of rows > 1\");\n }\n connection.commit();\n\ | |
| \ } catch (SQLException e) {\n Log.write(e.getMessage());\n\ | |
| \ throw new DAOException(\"A SQLException has occured\");\n \ | |
| \ } catch (NullPointerException npe) {\n Log.write(npe.getMessage());\n\ | |
| \ throw new DAOException(\"Connection null\");\n }\n \ | |
| \ return delete;\n }" | |
| - "public static String encrypt(String plaintext) throws EncryptionException {\n\ | |
| \ if (plaintext == null) {\n throw new EncryptionException();\n\ | |
| \ }\n try {\n final MessageDigest md = MessageDigest.getInstance(\"\ | |
| SHA\");\n md.update(plaintext.getBytes(\"UTF-8\"));\n return\ | |
| \ Base64.encodeBytes(md.digest());\n } catch (NoSuchAlgorithmException\ | |
| \ e) {\n throw new EncryptionException(e);\n } catch (UnsupportedEncodingException\ | |
| \ e) {\n throw new EncryptionException(e);\n }\n }" | |
| - "public static void copyFile(File in, File out) throws IOException {\n \ | |
| \ FileChannel sourceChannel = new FileInputStream(in).getChannel();\n FileChannel\ | |
| \ destinationChannel = new FileOutputStream(out).getChannel();\n destinationChannel.transferFrom(sourceChannel,\ | |
| \ 0, sourceChannel.size());\n sourceChannel.close();\n destinationChannel.close();\n\ | |
| \ }" | |
| - source_sentence: "private void sortWhats(String[] labels, int[] whats, String simplifyString)\ | |
| \ {\n int n = whats.length;\n boolean swapped;\n do {\n \ | |
| \ swapped = false;\n for (int i = 0; i < n - 1; i++) {\n\ | |
| \ int i0_pos = simplifyString.indexOf(labels[whats[i]]);\n \ | |
| \ int i1_pos = simplifyString.indexOf(labels[whats[i + 1]]);\n \ | |
| \ if (i0_pos > i1_pos) {\n int temp = whats[i];\n\ | |
| \ whats[i] = whats[i + 1];\n whats[i + 1]\ | |
| \ = temp;\n swapped = true;\n }\n \ | |
| \ }\n } while (swapped);\n }" | |
| sentences: | |
| - "public String GetMemberName(String id) {\n String name = null;\n \ | |
| \ try {\n String line;\n URL url = new URL(intvasmemberDeatails\ | |
| \ + \"?CID=\" + id);\n URLConnection connection = url.openConnection();\n\ | |
| \ BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));\n\ | |
| \ while ((line = reader.readLine()) != null) {\n name\ | |
| \ = line;\n }\n } catch (MalformedURLException e) {\n \ | |
| \ e.printStackTrace();\n } catch (IOException e) {\n e.printStackTrace();\n\ | |
| \ }\n String[] parts = name.split(\" \");\n rating = parts[2];\n\ | |
| \ return parts[0] + \" \" + parts[1];\n }" | |
| - "private static void readAndRewrite(File inFile, File outFile) throws IOException\ | |
| \ {\n ImageInputStream iis = ImageIO.createImageInputStream(new BufferedInputStream(new\ | |
| \ FileInputStream(inFile)));\n DcmParser dcmParser = DcmParserFactory.getInstance().newDcmParser(iis);\n\ | |
| \ Dataset ds = DcmObjectFactory.getInstance().newDataset();\n dcmParser.setDcmHandler(ds.getDcmHandler());\n\ | |
| \ dcmParser.parseDcmFile(null, Tags.PixelData);\n PixelDataReader\ | |
| \ pdReader = pdFact.newReader(ds, iis, dcmParser.getDcmDecodeParam().byteOrder,\ | |
| \ dcmParser.getReadVR());\n System.out.println(\"reading \" + inFile +\ | |
| \ \"...\");\n pdReader.readPixelData(false);\n ImageOutputStream\ | |
| \ out = ImageIO.createImageOutputStream(new BufferedOutputStream(new FileOutputStream(outFile)));\n\ | |
| \ DcmEncodeParam dcmEncParam = DcmEncodeParam.IVR_LE;\n ds.writeDataset(out,\ | |
| \ dcmEncParam);\n ds.writeHeader(out, dcmEncParam, Tags.PixelData, dcmParser.getReadVR(),\ | |
| \ dcmParser.getReadLength());\n System.out.println(\"writing \" + outFile\ | |
| \ + \"...\");\n PixelDataWriter pdWriter = pdFact.newWriter(pdReader.getPixelDataArray(),\ | |
| \ false, ds, out, dcmParser.getDcmDecodeParam().byteOrder, dcmParser.getReadVR());\n\ | |
| \ pdWriter.writePixelData();\n out.flush();\n out.close();\n\ | |
| \ System.out.println(\"done!\");\n }" | |
| - "public void insertComponent() throws SQLException {\n Connection connection\ | |
| \ = null;\n PreparedStatement ps = null;\n try {\n Class.forName(\"\ | |
| com.mysql.jdbc.Driver\");\n connection = (Connection) DriverManager.getConnection(this.jdbcURL);\n\ | |
| \ connection.setAutoCommit(false);\n String query = \"INSERT\ | |
| \ INTO components(name,rate,quantity, description) VALUES(?,?,?,?)\";\n \ | |
| \ ps = (PreparedStatement) connection.prepareStatement(query);\n \ | |
| \ ps.setString(1, this.name);\n ps.setDouble(2, this.rate);\n \ | |
| \ ps.setInt(3, this.quantity);\n ps.setString(4, this.description);\n\ | |
| \ ps.executeUpdate();\n connection.commit();\n }\ | |
| \ catch (Exception ex) {\n connection.rollback();\n } finally\ | |
| \ {\n try {\n connection.close();\n } catch\ | |
| \ (Exception ex) {\n }\n try {\n ps.close();\n\ | |
| \ } catch (Exception ex) {\n }\n }\n }" | |
| - source_sentence: "public static String toMD5(String seed) {\n MessageDigest\ | |
| \ md5 = null;\n StringBuffer temp_sb = null;\n try {\n \ | |
| \ md5 = MessageDigest.getInstance(\"MD5\");\n md5.update(seed.getBytes());\n\ | |
| \ byte[] array = md5.digest();\n temp_sb = new StringBuffer();\n\ | |
| \ for (int i = 0; i < array.length; i++) {\n int b =\ | |
| \ array[i] & 0xFF;\n if (b < 0x10) temp_sb.append('0');\n \ | |
| \ temp_sb.append(Integer.toHexString(b));\n }\n }\ | |
| \ catch (NoSuchAlgorithmException err) {\n err.printStackTrace();\n\ | |
| \ }\n return temp_sb.toString();\n }" | |
| sentences: | |
| - "public static boolean checkEncryptedPassword(String md5key, String passwordAccount,\ | |
| \ String encryptedPassword, int passwdenc) throws NoSuchAlgorithmException, UnsupportedEncodingException\ | |
| \ {\n MessageDigest md = MessageDigest.getInstance(Constants.ALGORITHM);\n\ | |
| \ switch(passwdenc) {\n case 1:\n md.update((md5key\ | |
| \ + encryptedPassword).getBytes(\"8859_1\"));\n return md.digest().equals(passwordAccount.getBytes(\"\ | |
| 8859_1\"));\n case 2:\n md.update((encryptedPassword\ | |
| \ + md5key).getBytes(\"8859_1\"));\n return md.digest().equals(passwordAccount.getBytes(\"\ | |
| 8859_1\"));\n default:\n return false;\n }\n\ | |
| \ }" | |
| - "@Test\n public void shouldDownloadFileUsingPublicLink() throws Exception {\n\ | |
| \ String bucketName = \"test-\" + UUID.randomUUID();\n Service service\ | |
| \ = new WebClientService(credentials);\n service.createBucket(bucketName);\n\ | |
| \ File file = folder.newFile(\"foo.txt\");\n FileUtils.writeStringToFile(file,\ | |
| \ UUID.randomUUID().toString());\n service.createObject(bucketName, file.getName(),\ | |
| \ file, new NullProgressListener());\n String publicUrl = service.getPublicUrl(bucketName,\ | |
| \ file.getName(), new DateTime().plusDays(5));\n File saved = folder.newFile(\"\ | |
| saved.txt\");\n InputStream input = new URL(publicUrl).openConnection().getInputStream();\n\ | |
| \ FileOutputStream output = new FileOutputStream(saved);\n IOUtils.copy(input,\ | |
| \ output);\n output.close();\n assertThat(\"Corrupted download\"\ | |
| , Files.computeMD5(saved), equalTo(Files.computeMD5(file)));\n service.deleteObject(bucketName,\ | |
| \ file.getName());\n service.deleteBucket(bucketName);\n }" | |
| - "public static void main(String[] args) {\n try {\n boolean\ | |
| \ readExp = Utils.getFlag('l', args);\n final boolean writeExp = Utils.getFlag('s',\ | |
| \ args);\n final String expFile = Utils.getOption('f', args);\n \ | |
| \ if ((readExp || writeExp) && (expFile.length() == 0)) {\n \ | |
| \ throw new Exception(\"A filename must be given with the -f option\");\n\ | |
| \ }\n Experiment exp = null;\n if (readExp) {\n\ | |
| \ FileInputStream fi = new FileInputStream(expFile);\n \ | |
| \ ObjectInputStream oi = new ObjectInputStream(new BufferedInputStream(fi));\n\ | |
| \ exp = (Experiment) oi.readObject();\n oi.close();\n\ | |
| \ } else {\n exp = new Experiment();\n }\n\ | |
| \ System.err.println(\"Initial Experiment:\\n\" + exp.toString());\n\ | |
| \ final JFrame jf = new JFrame(\"Weka Experiment Setup\");\n \ | |
| \ jf.getContentPane().setLayout(new BorderLayout());\n final SetupPanel\ | |
| \ sp = new SetupPanel();\n jf.getContentPane().add(sp, BorderLayout.CENTER);\n\ | |
| \ jf.addWindowListener(new WindowAdapter() {\n\n public\ | |
| \ void windowClosing(WindowEvent e) {\n System.err.println(\"\ | |
| \\nFinal Experiment:\\n\" + sp.m_Exp.toString());\n if (writeExp)\ | |
| \ {\n try {\n FileOutputStream\ | |
| \ fo = new FileOutputStream(expFile);\n ObjectOutputStream\ | |
| \ oo = new ObjectOutputStream(new BufferedOutputStream(fo));\n \ | |
| \ oo.writeObject(sp.m_Exp);\n oo.close();\n\ | |
| \ } catch (Exception ex) {\n \ | |
| \ ex.printStackTrace();\n System.err.println(\"Couldn't\ | |
| \ write experiment to: \" + expFile + '\\n' + ex.getMessage());\n \ | |
| \ }\n }\n jf.dispose();\n \ | |
| \ System.exit(0);\n }\n });\n \ | |
| \ jf.pack();\n jf.setVisible(true);\n System.err.println(\"\ | |
| Short nap\");\n Thread.currentThread().sleep(3000);\n System.err.println(\"\ | |
| Done\");\n sp.setExperiment(exp);\n } catch (Exception ex) {\n\ | |
| \ ex.printStackTrace();\n System.err.println(ex.getMessage());\n\ | |
| \ }\n }" | |
| - source_sentence: "protected void serveStaticContent(HttpServletRequest request,\ | |
| \ HttpServletResponse response, String pathInfo) throws ServletException {\n \ | |
| \ InputStream is = servletConfig.getServletContext().getResourceAsStream(pathInfo);\n\ | |
| \ if (is == null) {\n throw new ServletException(\"Static resource\ | |
| \ \" + pathInfo + \" is not available\");\n }\n try {\n \ | |
| \ int ind = pathInfo.lastIndexOf(\".\");\n if (ind != -1 && ind\ | |
| \ < pathInfo.length()) {\n String type = STATIC_CONTENT_TYPES.get(pathInfo.substring(ind\ | |
| \ + 1));\n if (type != null) {\n response.setContentType(type);\n\ | |
| \ }\n }\n ServletOutputStream os = response.getOutputStream();\n\ | |
| \ IOUtils.copy(is, os);\n os.flush();\n } catch (IOException\ | |
| \ ex) {\n throw new ServletException(\"Static resource \" + pathInfo\ | |
| \ + \" can not be written to the output stream\");\n }\n }" | |
| sentences: | |
| - "private static String makeLoginRequest(String authenticationUrl, String accountEmail,\ | |
| \ String accountPassword) throws IOException {\n URL url = new URL(authenticationUrl);\n\ | |
| \ HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();\n\ | |
| \ urlConnection.setRequestMethod(\"POST\");\n urlConnection.setDoInput(true);\n\ | |
| \ urlConnection.setDoOutput(true);\n urlConnection.setUseCaches(false);\n\ | |
| \ urlConnection.setRequestProperty(\"Content-Type\", \"application/x-www-form-urlencoded\"\ | |
| );\n StringBuffer content = new StringBuffer();\n content.append(\"\ | |
| Email=\").append(URLEncoder.encode(accountEmail, \"UTF-8\"));\n content.append(\"\ | |
| &Passwd=\").append(URLEncoder.encode(accountPassword, \"UTF-8\"));\n content.append(\"\ | |
| &source=\").append(URLEncoder.encode(\"Google Base data API for OFBiz\", \"UTF-8\"\ | |
| ));\n content.append(\"&service=\").append(URLEncoder.encode(\"gbase\"\ | |
| , \"UTF-8\"));\n OutputStream outputStream = urlConnection.getOutputStream();\n\ | |
| \ outputStream.write(content.toString().getBytes(\"UTF-8\"));\n \ | |
| \ outputStream.close();\n int responseCode = urlConnection.getResponseCode();\n\ | |
| \ InputStream inputStream;\n if (responseCode == HttpURLConnection.HTTP_OK)\ | |
| \ {\n inputStream = urlConnection.getInputStream();\n } else\ | |
| \ {\n inputStream = urlConnection.getErrorStream();\n }\n \ | |
| \ return toString(inputStream);\n }" | |
| - "public void constructFundamentalView() {\n String className;\n \ | |
| \ String methodName;\n String field;\n boolean foundRead = false;\n\ | |
| \ boolean foundWrite = false;\n boolean classWritten = false;\n\ | |
| \ try {\n FundView = new BufferedWriter(new FileWriter(\"InfoFiles/FundamentalView.txt\"\ | |
| ));\n FileInputStream fstreamPC = new FileInputStream(\"InfoFiles/PrincipleClassGroup.txt\"\ | |
| );\n DataInputStream inPC = new DataInputStream(fstreamPC);\n \ | |
| \ BufferedReader PC = new BufferedReader(new InputStreamReader(inPC));\n\ | |
| \ while ((field = PC.readLine()) != null) {\n className\ | |
| \ = field;\n FundView.write(className);\n FundView.newLine();\n\ | |
| \ classWritten = true;\n while ((methodName = PC.readLine())\ | |
| \ != null) {\n if (methodName.contentEquals(\"EndOfClass\"\ | |
| )) break;\n FundView.write(\"StartOfMethod\");\n \ | |
| \ FundView.newLine();\n FundView.write(methodName);\n\ | |
| \ FundView.newLine();\n for (int i = 0;\ | |
| \ i < readFileCount && foundRead == false; i++) {\n if\ | |
| \ (methodName.compareTo(readArray[i]) == 0) {\n foundRead\ | |
| \ = true;\n for (int j = 1; readArray[i + j].compareTo(\"\ | |
| EndOfMethod\") != 0; j++) {\n if (readArray[i +\ | |
| \ j].indexOf(\".\") > 0) {\n field = readArray[i\ | |
| \ + j].substring(0, readArray[i + j].indexOf(\".\"));\n \ | |
| \ if (field.compareTo(className) == 0) {\n \ | |
| \ FundView.write(readArray[i + j]);\n \ | |
| \ FundView.newLine();\n }\n\ | |
| \ }\n }\n \ | |
| \ }\n }\n for (int i = 0; i\ | |
| \ < writeFileCount && foundWrite == false; i++) {\n if\ | |
| \ (methodName.compareTo(writeArray[i]) == 0) {\n foundWrite\ | |
| \ = true;\n for (int j = 1; writeArray[i + j].compareTo(\"\ | |
| EndOfMethod\") != 0; j++) {\n if (writeArray[i\ | |
| \ + j].indexOf(\".\") > 0) {\n field = writeArray[i\ | |
| \ + j].substring(0, writeArray[i + j].indexOf(\".\"));\n \ | |
| \ if (field.compareTo(className) == 0) {\n \ | |
| \ FundView.write(writeArray[i + j]);\n \ | |
| \ FundView.newLine();\n }\n\ | |
| \ }\n }\n \ | |
| \ }\n }\n FundView.write(\"\ | |
| EndOfMethod\");\n FundView.newLine();\n \ | |
| \ foundRead = false;\n foundWrite = false;\n \ | |
| \ }\n if (classWritten == true) {\n FundView.write(\"\ | |
| EndOfClass\");\n FundView.newLine();\n classWritten\ | |
| \ = false;\n }\n }\n PC.close();\n \ | |
| \ FundView.close();\n } catch (IOException e) {\n e.printStackTrace();\n\ | |
| \ }\n }" | |
| - "public static void main(String[] args) throws Exception {\n if (args.length\ | |
| \ != 2) {\n System.out.println(\"arguments: sourcefile destfile\");\n\ | |
| \ System.exit(1);\n }\n FileChannel in = new FileInputStream(args[0]).getChannel(),\ | |
| \ out = new FileOutputStream(args[1]).getChannel();\n in.transferTo(0,\ | |
| \ in.size(), out);\n }" | |
| pipeline_tag: sentence-similarity | |
| library_name: sentence-transformers | |
| metrics: | |
| - pearson_cosine | |
| - spearman_cosine | |
| model-index: | |
| - name: SentenceTransformer based on microsoft/codebert-base | |
| results: | |
| - task: | |
| type: semantic-similarity | |
| name: Semantic Similarity | |
| dataset: | |
| name: bigclonebench validation | |
| type: bigclonebench-validation | |
| metrics: | |
| - type: pearson_cosine | |
| value: 0.9050117464281175 | |
| name: Pearson Cosine | |
| - type: spearman_cosine | |
| value: 0.8342891483192489 | |
| name: Spearman Cosine | |
| # SentenceTransformer based on microsoft/codebert-base | |
| This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more. | |
| ## Model Details | |
| ### Model Description | |
| - **Model Type:** Sentence Transformer | |
| - **Base model:** [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) <!-- at revision 3b0952feddeffad0063f274080e3c23d75e7eb39 --> | |
| - **Maximum Sequence Length:** 256 tokens | |
| - **Output Dimensionality:** 768 dimensions | |
| - **Similarity Function:** Cosine Similarity | |
| - **Supported Modality:** Text | |
| <!-- - **Training Dataset:** Unknown --> | |
| <!-- - **Language:** Unknown --> | |
| <!-- - **License:** Unknown --> | |
| ### Model Sources | |
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) | |
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) | |
| - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) | |
| ### Full Model Architecture | |
| ``` | |
| SentenceTransformer( | |
| (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RobertaModel'}) | |
| (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True}) | |
| ) | |
| ``` | |
| ## Usage | |
| ### Direct Usage (Sentence Transformers) | |
| First install the Sentence Transformers library: | |
| ```bash | |
| pip install -U sentence-transformers | |
| ``` | |
| Then you can load this model and run inference. | |
| ```python | |
| from sentence_transformers import SentenceTransformer | |
| # Download from the 🤗 Hub | |
| model = SentenceTransformer("wraithklu/codesentinel-clone-detector") | |
| # Run inference | |
| sentences = [ | |
| 'protected void serveStaticContent(HttpServletRequest request, HttpServletResponse response, String pathInfo) throws ServletException {\n InputStream is = servletConfig.getServletContext().getResourceAsStream(pathInfo);\n if (is == null) {\n throw new ServletException("Static resource " + pathInfo + " is not available");\n }\n try {\n int ind = pathInfo.lastIndexOf(".");\n if (ind != -1 && ind < pathInfo.length()) {\n String type = STATIC_CONTENT_TYPES.get(pathInfo.substring(ind + 1));\n if (type != null) {\n response.setContentType(type);\n }\n }\n ServletOutputStream os = response.getOutputStream();\n IOUtils.copy(is, os);\n os.flush();\n } catch (IOException ex) {\n throw new ServletException("Static resource " + pathInfo + " can not be written to the output stream");\n }\n }', | |
| 'private static String makeLoginRequest(String authenticationUrl, String accountEmail, String accountPassword) throws IOException {\n URL url = new URL(authenticationUrl);\n HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();\n urlConnection.setRequestMethod("POST");\n urlConnection.setDoInput(true);\n urlConnection.setDoOutput(true);\n urlConnection.setUseCaches(false);\n urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");\n StringBuffer content = new StringBuffer();\n content.append("Email=").append(URLEncoder.encode(accountEmail, "UTF-8"));\n content.append("&Passwd=").append(URLEncoder.encode(accountPassword, "UTF-8"));\n content.append("&source=").append(URLEncoder.encode("Google Base data API for OFBiz", "UTF-8"));\n content.append("&service=").append(URLEncoder.encode("gbase", "UTF-8"));\n OutputStream outputStream = urlConnection.getOutputStream();\n outputStream.write(content.toString().getBytes("UTF-8"));\n outputStream.close();\n int responseCode = urlConnection.getResponseCode();\n InputStream inputStream;\n if (responseCode == HttpURLConnection.HTTP_OK) {\n inputStream = urlConnection.getInputStream();\n } else {\n inputStream = urlConnection.getErrorStream();\n }\n return toString(inputStream);\n }', | |
| 'public static void main(String[] args) throws Exception {\n if (args.length != 2) {\n System.out.println("arguments: sourcefile destfile");\n System.exit(1);\n }\n FileChannel in = new FileInputStream(args[0]).getChannel(), out = new FileOutputStream(args[1]).getChannel();\n in.transferTo(0, in.size(), out);\n }', | |
| ] | |
| embeddings = model.encode(sentences) | |
| print(embeddings.shape) | |
| # [3, 768] | |
| # Get the similarity scores for the embeddings | |
| similarities = model.similarity(embeddings, embeddings) | |
| print(similarities) | |
| # tensor([[ 1.0000, -0.0114, 0.9605], | |
| # [-0.0114, 1.0000, 0.0068], | |
| # [ 0.9605, 0.0068, 1.0000]]) | |
| ``` | |
| <!-- | |
| ### Direct Usage (Transformers) | |
| <details><summary>Click to see the direct usage in Transformers</summary> | |
| </details> | |
| --> | |
| <!-- | |
| ### Downstream Usage (Sentence Transformers) | |
| You can finetune this model on your own dataset. | |
| <details><summary>Click to expand</summary> | |
| </details> | |
| --> | |
| <!-- | |
| ### Out-of-Scope Use | |
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* | |
| --> | |
| ## Evaluation | |
| ### Metrics | |
| #### Semantic Similarity | |
| * Dataset: `bigclonebench-validation` | |
| * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.sentence_transformer.evaluation.EmbeddingSimilarityEvaluator) | |
| | Metric | Value | | |
| |:--------------------|:-----------| | |
| | pearson_cosine | 0.905 | | |
| | **spearman_cosine** | **0.8343** | | |
| <!-- | |
| ## Bias, Risks and Limitations | |
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* | |
| --> | |
| <!-- | |
| ### Recommendations | |
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* | |
| --> | |
| ## Training Details | |
| ### Training Dataset | |
| #### Unnamed Dataset | |
| * Size: 10,000 training samples | |
| * Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>label</code> | |
| * Approximate statistics based on the first 100 samples: | |
| | | sentence_0 | sentence_1 | label | | |
| |:---------|:--------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|:---------------------------------------------------------------| | |
| | type | string | string | float | | |
| | modality | text | text | | | |
| | details | <ul><li>min: 134 tokens</li><li>mean: 240.02 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 104 tokens</li><li>mean: 237.68 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.49</li><li>max: 1.0</li></ul> | | |
| * Samples: | |
| | sentence_0 | sentence_1 | label | | |
| |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------| | |
| | <code>public InputLattice(String file, int type, boolean convertJoinMeet) throws FileNotFoundException, IOException {<br> InputStream latStream = null;<br> labels = new Vector();<br> upperCoversVector = new Vector();<br> Vector upperCovers = new Vector();<br> edgeColors = new Hashtable();<br> Vector upperCovering = new Vector();<br> int level = 0;<br> String str;<br> String currentElt = null;<br> String cover = null;<br> boolean first = true;<br> StreamTokenizer in;<br> switch(type) {<br> case FILE:<br> latStream = new FileInputStream(file);<br> break;<br> case STRING:<br> latStream = new StringBufferInputStream(file);<br> break;<br> case URL:<br> URL url = new URL("http", Config.HOST, Config.PORT, "/" + Config.HASSE + "/" + Config.URL_STREAMER + file);<br> System.out.println("url is " + url);<br> URLConnection connection = url.openCo...</code> | <code>private String calculateHash(String s) {<br> if (s == null) {<br> return null;<br> }<br> MessageDigest messageDigest;<br> try {<br> messageDigest = MessageDigest.getInstance("SHA");<br> } catch (NoSuchAlgorithmException e) {<br> logger.error("Could not find a message digest algorithm.");<br> return null;<br> }<br> messageDigest.update(s.getBytes());<br> byte[] hash = messageDigest.digest();<br> StringBuilder sb = new StringBuilder();<br> for (byte b : hash) {<br> sb.append(String.format("%02x", b));<br> }<br> return sb.toString();<br> }</code> | <code>0.0</code> | | |
| | <code>void load(URL url) throws IOException {<br> BufferedReader r = new BufferedReader(new InputStreamReader(url.openStream()));<br> Vector3f scale = new Vector3f(1, 1, 1);<br> Group currentGroup = new Group();<br> currentGroup.name = "default";<br> groups.add(currentGroup);<br> String line;<br> while ((line = r.readLine()) != null) {<br> String[] params = line.split(" +");<br> if (params.length == 0) continue;<br> String command = params[0];<br> if (params[0].equals("v")) {<br> Vector3f vertex = new Vector3f(Float.parseFloat(params[1]) * scale.x, Float.parseFloat(params[2]) * scale.y, Float.parseFloat(params[3]) * scale.z);<br> verticies.add(vertex);<br> radius = Math.max(radius, vertex.length());<br> }<br> if (command.equals("center")) {<br> epicenter = new Vector3f(Float.parseFloat(params[1]), Float.parseFloat(params[2]), Float.parseFloat(params[3]));<br> } ...</code> | <code>public static boolean canWeConnectToInternet() {<br> String s = "http://www.google.com/";<br> URL url = null;<br> boolean can = false;<br> URLConnection conection = null;<br> try {<br> url = new URL(s);<br> } catch (MalformedURLException e) {<br> System.out.println("This should never happend. Error in URL name. URL specified was:" + s + ".");<br> }<br> try {<br> conection = url.openConnection();<br> conection.connect();<br> can = true;<br> } catch (IOException e) {<br> can = false;<br> }<br> if (can) {<br> }<br> return can;<br> }</code> | <code>0.0</code> | | |
| | <code>public static java.io.ByteArrayOutputStream getFileByteStream(URL _url) {<br> java.io.ByteArrayOutputStream buffer = new java.io.ByteArrayOutputStream();<br> try {<br> InputStream input = _url.openStream();<br> IOUtils.copy(input, buffer);<br> IOUtils.closeQuietly(input);<br> } catch (Exception err) {<br> throw new RuntimeException(err);<br> }<br> return buffer;<br> }</code> | <code>public BufferedImage processUsingTemp(InputStream input, DjatokaDecodeParam params) throws DjatokaException {<br> File in;<br> try {<br> in = File.createTempFile("tmp", ".jp2");<br> FileOutputStream fos = new FileOutputStream(in);<br> in.deleteOnExit();<br> IOUtils.copyStream(input, fos);<br> } catch (IOException e) {<br> logger.error(e, e);<br> throw new DjatokaException(e);<br> }<br> BufferedImage bi = process(in.getAbsolutePath(), params);<br> if (in != null) in.delete();<br> return bi;<br> }</code> | <code>1.0</code> | | |
| * Loss: [<code>CosineSimilarityLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosinesimilarityloss) with these parameters: | |
| ```json | |
| { | |
| "loss_fct": "torch.nn.modules.loss.MSELoss", | |
| "cos_score_transformation": "torch.nn.modules.linear.Identity" | |
| } | |
| ``` | |
| ### Training Hyperparameters | |
| #### Non-Default Hyperparameters | |
| - `per_device_train_batch_size`: 32 | |
| - `num_train_epochs`: 2 | |
| - `fp16`: True | |
| - `per_device_eval_batch_size`: 32 | |
| - `multi_dataset_batch_sampler`: round_robin | |
| #### All Hyperparameters | |
| <details><summary>Click to expand</summary> | |
| - `per_device_train_batch_size`: 32 | |
| - `num_train_epochs`: 2 | |
| - `max_steps`: -1 | |
| - `learning_rate`: 5e-05 | |
| - `lr_scheduler_type`: linear | |
| - `lr_scheduler_kwargs`: None | |
| - `warmup_steps`: 0 | |
| - `optim`: adamw_torch_fused | |
| - `optim_args`: None | |
| - `weight_decay`: 0.0 | |
| - `adam_beta1`: 0.9 | |
| - `adam_beta2`: 0.999 | |
| - `adam_epsilon`: 1e-08 | |
| - `optim_target_modules`: None | |
| - `gradient_accumulation_steps`: 1 | |
| - `average_tokens_across_devices`: True | |
| - `max_grad_norm`: 1 | |
| - `label_smoothing_factor`: 0.0 | |
| - `bf16`: False | |
| - `fp16`: True | |
| - `bf16_full_eval`: False | |
| - `fp16_full_eval`: False | |
| - `tf32`: None | |
| - `gradient_checkpointing`: False | |
| - `gradient_checkpointing_kwargs`: None | |
| - `torch_compile`: False | |
| - `torch_compile_backend`: None | |
| - `torch_compile_mode`: None | |
| - `use_liger_kernel`: False | |
| - `liger_kernel_config`: None | |
| - `use_cache`: False | |
| - `neftune_noise_alpha`: None | |
| - `torch_empty_cache_steps`: None | |
| - `auto_find_batch_size`: False | |
| - `log_on_each_node`: True | |
| - `logging_nan_inf_filter`: True | |
| - `include_num_input_tokens_seen`: no | |
| - `log_level`: passive | |
| - `log_level_replica`: warning | |
| - `disable_tqdm`: False | |
| - `project`: huggingface | |
| - `trackio_space_id`: None | |
| - `trackio_bucket_id`: None | |
| - `trackio_static_space_id`: None | |
| - `per_device_eval_batch_size`: 32 | |
| - `prediction_loss_only`: True | |
| - `eval_on_start`: False | |
| - `eval_do_concat_batches`: True | |
| - `eval_use_gather_object`: False | |
| - `eval_accumulation_steps`: None | |
| - `include_for_metrics`: [] | |
| - `batch_eval_metrics`: False | |
| - `save_only_model`: False | |
| - `save_on_each_node`: False | |
| - `enable_jit_checkpoint`: False | |
| - `push_to_hub`: False | |
| - `hub_private_repo`: None | |
| - `hub_model_id`: None | |
| - `hub_strategy`: every_save | |
| - `hub_always_push`: False | |
| - `hub_revision`: None | |
| - `load_best_model_at_end`: False | |
| - `ignore_data_skip`: False | |
| - `restore_callback_states_from_checkpoint`: False | |
| - `full_determinism`: False | |
| - `seed`: 42 | |
| - `data_seed`: None | |
| - `use_cpu`: False | |
| - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} | |
| - `parallelism_config`: None | |
| - `dataloader_drop_last`: False | |
| - `dataloader_num_workers`: 0 | |
| - `dataloader_pin_memory`: True | |
| - `dataloader_persistent_workers`: False | |
| - `dataloader_prefetch_factor`: None | |
| - `dataloader_multiprocessing_context`: None | |
| - `dataloader_in_order`: True | |
| - `remove_unused_columns`: True | |
| - `label_names`: None | |
| - `train_sampling_strategy`: random | |
| - `length_column_name`: length | |
| - `ddp_find_unused_parameters`: None | |
| - `ddp_bucket_cap_mb`: None | |
| - `ddp_broadcast_buffers`: False | |
| - `ddp_static_graph`: None | |
| - `ddp_backend`: None | |
| - `ddp_timeout`: 1800 | |
| - `fsdp`: None | |
| - `fsdp_config`: None | |
| - `deepspeed`: None | |
| - `debug`: [] | |
| - `skip_memory_metrics`: True | |
| - `do_predict`: False | |
| - `resume_from_checkpoint`: None | |
| - `local_rank`: -1 | |
| - `prompts`: None | |
| - `batch_sampler`: batch_sampler | |
| - `multi_dataset_batch_sampler`: round_robin | |
| - `router_mapping`: {} | |
| - `learning_rate_mapping`: {} | |
| - `warmup_ratio`: None | |
| </details> | |
| ### Training Logs | |
| | Epoch | Step | Training Loss | bigclonebench-validation_spearman_cosine | | |
| |:------:|:----:|:-------------:|:----------------------------------------:| | |
| | 0.4984 | 156 | - | 0.8142 | | |
| | 0.9968 | 312 | - | 0.8325 | | |
| | 1.0 | 313 | - | 0.8325 | | |
| | 1.4952 | 468 | - | 0.8346 | | |
| | 1.5974 | 500 | 0.0901 | - | | |
| | 1.9936 | 624 | - | 0.8343 | | |
| | 2.0 | 626 | - | 0.8343 | | |
| ### Training Time | |
| - **Training**: 13.3 minutes | |
| ### Framework Versions | |
| - Python: 3.13.16 | |
| - Sentence Transformers: 5.7.0 | |
| - Transformers: 5.22.1 | |
| - PyTorch: 2.14.1+cu130 | |
| - Accelerate: 1.15.0 | |
| - Datasets: 4.8.5 | |
| - Tokenizers: 0.23.3 | |
| ## Additional Resources | |
| - [Training and Finetuning Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sentence-transformers): the end-to-end guide for training or finetuning Sentence Transformer models. | |
| - [Introduction to Matryoshka Embedding Models](https://huggingface.co/blog/matryoshka): variable-size embeddings that can be truncated with minimal quality loss. | |
| - [Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval](https://huggingface.co/blog/embedding-quantization): post-training compression of embedding vectors. | |
| - [Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/multimodal-sentence-transformers): use text, image, audio, and video models through the same API. | |
| - [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): train multimodal embedding models, with a Visual Document Retrieval walkthrough. | |
| ## Citation | |
| ### BibTeX | |
| #### Sentence Transformers | |
| ```bibtex | |
| @inproceedings{reimers-2019-sentence-bert, | |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", | |
| author = "Reimers, Nils and Gurevych, Iryna", | |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", | |
| month = "11", | |
| year = "2019", | |
| publisher = "Association for Computational Linguistics", | |
| url = "https://arxiv.org/abs/1908.10084", | |
| } | |
| ``` | |
| <!-- | |
| ## Glossary | |
| *Clearly define terms in order to be accessible across audiences.* | |
| --> | |
| <!-- | |
| ## Model Card Authors | |
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* | |
| --> | |
| <!-- | |
| ## Model Card Contact | |
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* | |
| --> |