Update README.md

README.md

@@ -1,92 +1,101 @@
 ---
-library_name: model2vec
+language:
+- code
 license: mit
-model_name: minishlab/potion-code-16M-v2
+library_name: model2vec
 tags:
+- model2vec
 - embeddings
+- code
+- retrieval
 - static-embeddings
-- sentence-transformers
+datasets:
+- minishlab/tokenlearn-cornstack-queries-coderankembed
+- minishlab/tokenlearn-cornstack-docs-coderankembed
+- nomic-ai/cornstack-python-v1
+- nomic-ai/cornstack-java-v1
+- nomic-ai/cornstack-php-v1
+- nomic-ai/cornstack-go-v1
+- nomic-ai/cornstack-javascript-v1
+- nomic-ai/cornstack-ruby-v1
 ---
 
-# minishlab/potion-code-16M-v2 Model Card
+# potion-code-16M-v2 Model Card
 
-This [Model2Vec](https://github.com/MinishLab/model2vec) model is a distilled version of a Sentence Transformer. It uses static embeddings, allowing text embeddings to be computed orders of magnitude faster on both GPU and CPU. It is designed for applications where computational resources are limited or where real-time performance is critical. Model2Vec models are the smallest, fastest, and most performant static embedders available. The distilled models are up to 50 times smaller and 500 times faster than traditional Sentence Transformers.
+## Overview
 
+**potion-code-16M-v2** is a fast static code embedding model optimized for code retrieval tasks. It powers [Semble](https://github.com/MinishLab/semble), a code search library for agents. It is distilled from [nomic-ai/CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) and trained on the [CornStack](https://huggingface.co/datasets/nomic-ai/cornstack-python-v1) code corpus using [Tokenlearn](https://github.com/MinishLab/tokenlearn) and contrastive fine-tuning.
+It is the successor to [potion-code-16M](https://huggingface.co/minishlab/potion-code-16M).
+It uses static embeddings, allowing text and code embeddings to be computed orders of magnitude faster than transformer-based models on both GPU and CPU.
 
 ## Installation
 
-Install model2vec using pip:
-```
+```bash
 pip install model2vec
 ```
 
 ## Usage
 
-### Using Model2Vec
-
-The [Model2Vec library](https://github.com/MinishLab/model2vec) is the fastest and most lightweight way to run Model2Vec models.
-
-Load this model using the `from_pretrained` method:
 ```python
 from model2vec import StaticModel
 
-# Load a pretrained Model2Vec model
 model = StaticModel.from_pretrained("minishlab/potion-code-16M-v2")
 
-# Compute text embeddings
-embeddings = model.encode(["Example sentence"])
-```
-
-### Using Sentence Transformers
-
-You can also use the [Sentence Transformers library](https://github.com/UKPLab/sentence-transformers) to load and use the model:
-
-```python
-from sentence_transformers import SentenceTransformer
-
-# Load a pretrained Sentence Transformer model
-model = SentenceTransformer("minishlab/potion-code-16M-v2")
+# Embed natural language queries
+query_embeddings = model.encode(["How to read a file in Python?"])
 
-# Compute text embeddings
-embeddings = model.encode(["Example sentence"])
+# Embed code documents
+code_embeddings = model.encode(["def read_file(path):\n    with open(path) as f:\n        return f.read()"])
 ```
 
-### Distilling a Model2Vec model
-
-You can distill a Model2Vec model from a Sentence Transformer model using the `distill` method. First, install the `distill` extra with `pip install model2vec[distill]`. Then, run the following code:
-
-```python
-from model2vec.distill import distill
+## How it works
 
-# Distill a Sentence Transformer model, in this case the BAAI/bge-base-en-v1.5 model
-m2v_model = distill(model_name="BAAI/bge-base-en-v1.5", pca_dims=256)
+potion-code-16M-v2 is created using the following pipeline:
 
-# Save the model
-m2v_model.save_pretrained("m2v_model")
-```
+1. **Vocabulary mining**: code-specific tokens are mined from CornStack and added to the base CodeRankEmbed tokenizer (43k extra tokens → ~63.5k total)
+2. **Distillation**: the extended vocabulary is distilled from CodeRankEmbed using Model2Vec (256-dimensional embeddings, PCA)
+3. **Tokenlearn**: the distilled model is fine-tuned on 1.2 million (query, document) pairs from CornStack using cosine similarity loss
+4. **Contrastive fine-tuning**: the model is further fine-tuned using MultipleNegativesRankingLoss on 1.2 million CornStack query-document pairs
 
-## How it works
+## Results
 
-Model2vec creates a small, fast, and powerful model that outperforms other static embedding models by a large margin on all tasks we could find, while being much faster to create than traditional static embedding models such as GloVe. Best of all, you don't need any data to distill a model using Model2Vec.
+Results on the [CoIR benchmark](https://github.com/CoIR-team/coir) on [MTEB](https://github.com/embeddings-benchmark/mteb) (NDCG@10, `mteb>=2.10`):
 
-It works by passing a vocabulary through a sentence transformer model, then reducing the dimensionality of the resulting embeddings using PCA, and finally weighting the embeddings using [SIF weighting](https://openreview.net/pdf?id=SyK00v5xx). During inference, we simply take the mean of all token embeddings occurring in a sentence.
+| Model | Params | AVG | AppsRetrieval | COIRCodeSearchNet | CodeFeedbackMT | CodeFeedbackST | CodeSearchNetCC | CodeTransContest | CodeTransDL | CosQA | StackOverflow | Text2SQL |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| CodeRankEmbed | 137M | 59.14 | 23.46 | 94.70 | 42.61 | 78.11 | 76.39 | 66.43 | 34.84 | 35.92 | 80.53 | 58.37 |
+| **potion-code-16M + Hybrid** | **16M** | **40.41** | **5.23** | **34.03** | **51.23** | **64.26** | **33.22** | **52.67** | **31.14** | **21.63** | **69.65** | **41.03** |
+| BM25 | — | 39.11 | 4.76 | 32.45 | 59.69 | 67.85 | 33.00 | 47.29 | 32.97 | 15.53 | 69.54 | 28.07 |
+| **potion-code-16M** | **16M** | **37.05** | **3.97** | **42.99** | **36.26** | **50.27** | **43.40** | **39.76** | **31.72** | **21.37** | **57.47** | **43.34** |
+| potion-retrieval-32M | 32M | 32.10 | 4.22 | 31.80 | 36.71 | 45.11 | 38.64 | 29.97 | 32.62 | 8.70 | 56.26 | 36.93 |
+| potion-base-32M | 32M | 31.42 | 3.37 | 29.58 | 34.77 | 42.69 | 37.88 | 28.51 | 30.55 | 14.61 | 53.36 | 38.88 |
 
-## Additional Resources
+CoIR covers a broad range of code retrieval scenarios. For the use case of finding code given a natural language query, **CosQA** and **CodeFeedback (ST/MT)** are the most relevant tasks. Others are less so: **COIRCodeSearchNetRetrieval** retrieves text given a code query (the reverse direction), and the **CodeTransOcean** tasks target cross-language code translation. The hybrid row combines dense retrieval with BM25 using min-max score normalization and equal weighting (alpha=0.5).
 
-- [Model2Vec Repo](https://github.com/MinishLab/model2vec)
-- [Model2Vec Base Models](https://huggingface.co/collections/minishlab/model2vec-base-models-66fd9dd9b7c3b3c0f25ca90e)
-- [Model2Vec Results](https://github.com/MinishLab/model2vec/tree/main/results)
-- [Model2Vec Docs](https://minish.ai/packages/model2vec/introduction)
+## Model Details
 
+| Property | Value |
+|---|---|
+| Parameters | ~16M |
+| Embedding dimensions | 256 |
+| Vocabulary size | ~63,500 |
+| Teacher model | nomic-ai/CodeRankEmbed |
+| Training corpus | CornStack (6 languages: Python, Java, JavaScript, Go, PHP, Ruby) |
+| Max sequence length | 1,000,000 tokens (static, no limit in practice) |
 
-## Library Authors
+## Additional Resources
 
-Model2Vec was developed by the [Minish Lab](https://github.com/MinishLab) team consisting of [Stephan Tulkens](https://github.com/stephantul) and [Thomas van Dongen](https://github.com/Pringled).
+- [Semble repository](https://github.com/MinishLab/semble)
+- [Model2Vec repository](https://github.com/MinishLab/model2vec)
+- [Tokenlearn repository](https://github.com/MinishLab/tokenlearn)
+- [Tokenlearn document dataset](https://huggingface.co/minishlab/tokenlearn-cornstack-docs-coderankembed-v2)
+- [Tokenlearn query dataset](https://huggingface.co/minishlab/tokenlearn-cornstack-queries-coderankembed-v2)
+- [CornStack dataset](https://huggingface.co/datasets/nomic-ai/cornstack-python-v1)
+- [CoIR benchmark](https://github.com/CoIR-team/coir)
 
 ## Citation
 
-Please cite the [Model2Vec repository](https://github.com/MinishLab/model2vec) if you use this model in your work.
-```
+```bibtex
 @software{minishlab2024model2vec,
   author       = {Stephan Tulkens and {van Dongen}, Thomas},
   title        = {Model2Vec: Fast State-of-the-Art Static Embeddings},
@@ -96,4 +105,4 @@ Please cite the [Model2Vec repository](https://github.com/MinishLab/model2vec) i
   url          = {https://github.com/MinishLab/model2vec},
   license      = {MIT}
 }
-```
+```