README.md

---
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: "Analyze this clean_code: perms = models.Permission.objects.filter(\n    \
    \        codename__in=(\"add_customuser\", \"change_customuser\")\n        )\n\
    \        user.user_permissions.add(*perms)\n        request = self.factory.get(\"\
    /rand\")\n        request.user = user"
- text: 'Analyze this hardcoded_secret: SECRET_KEY = "super-secret-jwt-key-do-not-share"  #
    Django production secret'
- text: "Analyze this vulnerable_pattern: var name_input = document.getElementById\
    \ ('name');\n\n\t\t\tif (user_json.name == '') {\n\t\t\t\tuser_info.innerHTML\
    \ = 'User details: unknown user';\n\t\t\t\tname_input.value = 'unknown';\n\t\t\
    \t} else {\n\t\t\t\tvar level = 'unknown';"
- text: 'Analyze this hardcoded_secret: const db = new Pool({ password: ''Pr0duct10n#2024''
    });'
- text: "Analyze this hardcoded_secret: name:        \"Blake2b with 'Hello, World!'\"\
    ,\n\t\t\thasher:      NewBlake2B(),\n\t\t\tinput:       []byte(\"Hello, World!\"\
    ),\n\t\t\texpectedHex: \"511bc81dde11180838c562c82bb35f3223f46061ebde4a955c27b3f489cf1e03\"\
    ,\n\t\t},\n\t\t{\n\t\t\tname:        \"Blake2b input at max size\","
metrics:
- accuracy
pipeline_tag: text-classification
library_name: setfit
inference: true
model-index:
- name: SetFit
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.9808612440191388
      name: Accuracy
---

# SetFit

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
<!-- - **Sentence Transformer:** [Unknown](https://huggingface.co/unknown) -->
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 256 tokens
- **Number of Classes:** 4 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label            | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|:-----------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| SAFE_CODE        | <ul><li>'Analyze this clean_code: },\n                        "summary": "Read Items",\n                        "operationId": "read_items_items__get",\n                        "security": [{"OAuth2PasswordBearer": []}],'</li><li>'Analyze this clean_code: :license: Apache 2.0, see LICENSE for more details.\n"""\n\nfrom __future__ import annotations\n\nimport warnings'</li><li>'Analyze this clean_code: it \'sets rel to stylesheet\' do\n      get \'/style\'\n      expect(headers[\'Link\']).to include(\'rel="stylesheet"\')\n    end\n\n    it \'returns html tag\' do'</li></ul>                                                                                                                                                                                                                                   |
| VULNERABLE_LOGIC | <ul><li>'try {\n        res = await exec(`${ezu} apikey-update --apikey user --attributes doi,is_oa`);'</li><li>'cls()\n        os.system("git clone https://github.com/mrwn007/M3M0")'</li><li>'count = cursor.fetchone()[0]\n\tcursor.execute("SELECT text FROM posts OFFSET random()*" + str(count) + " LIMIT 1;")'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| REAL_SECRET      | <ul><li>'Analyze this hardcoded_secret: OPENAI_API_KEY=sk-proj-Tz9mK2nP7qR4sT6uW8vE5fG'</li><li>'Analyze this hardcoded_secret: $this->data = array (\n\t\t\t1 => new User (1, "tony", 0, \'1c8bfe8f801d79745c4631d09fff36c82aa37fc4cce4fc946683d7b336b63032\'),\n\t\t\t2 => new User (2, "morph", 1, \'e5326ba4359f77c2623244acb04f6ac35c4dfca330ebcccdf9b734e5b1df90a8\'),\n\t\t\t3 => new User (3, "chas", 1, \'a89237fc1f9dd8d424d8b8b98b890dbc4a817bfde59af17c39debcc4a14c21de\'),\n\t\t);\n\t\t$this->requestMethod = $requestMethod;\n\t\t$this->userId = $userId;'</li><li>"openai_key: ['Use sk-abcdefghijklmnopqrstuvwxyz0123456789 for calls', 'sk-abcdefghijklmnopqrstuvwxyz0123456789'],\n  github_token: ['Push with ghp_abcdefghijklmnopqrstuvwxyz0123456789', 'ghp_abcdefghijklmnopqrstuvwxyz0123456789'],"</li></ul> |
| TEST_MOCK        | <ul><li>'Analyze this test_fixture: expected2 = "y=b\'\\\\xe2\\\\xe1\', x=b\'\\\\xe1\\\\xe2\'"\n        self.assertIn(txt, (expected1, expected2))\n\n    def test_text_kwargs(self):\n        txt = str(utils.log_pprint(kwargs={\'x\': \'ŧêßŧ\', \'y\': \'ŧßêŧ\'}))\n        expected1 = "x=\'ŧêßŧ\', y=\'ŧßêŧ\'"\n        expected2 = "y=\'ŧßêŧ\', x=\'ŧêßŧ\'"'</li><li>'const ACCESS_TOKEN = "AKIAIOSFODNN7EXAMPLE"'</li><li>"Analyze this test_fixture: os.pardir,                                      # /path/to/fboy\n)\n\nMEDIA_ROOT = os.path.join(FACTORY_ROOT, 'tmp_test')\n\nDATABASES = {\n    'default': {"</li></ul>                                                                                                                                                                                                  |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.9809   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Analyze this hardcoded_secret: const db = new Pool({ password: 'Pr0duct10n#2024' });")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 1   | 22.5510 | 224 |

| Label            | Training Sample Count |
|:-----------------|:----------------------|
| REAL_SECRET      | 113                   |
| VULNERABLE_LOGIC | 240                   |
| TEST_MOCK        | 240                   |
| SAFE_CODE        | 240                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 5
- body_learning_rate: (2e-05, 1e-05)
- head_learning_rate: 0.01
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: True

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0019 | 1    | 0.0013        | -               |
| 0.0960 | 50   | 0.0125        | -               |
| 0.1919 | 100  | 0.0052        | -               |
| 0.2879 | 150  | 0.0097        | -               |
| 0.3839 | 200  | 0.0039        | -               |
| 0.4798 | 250  | 0.0032        | -               |
| 0.5758 | 300  | 0.0015        | -               |
| 0.6718 | 350  | 0.0016        | -               |
| 0.7678 | 400  | 0.0011        | -               |
| 0.8637 | 450  | 0.0013        | -               |
| 0.9597 | 500  | 0.0022        | -               |
| 1.0    | 521  | -             | 0.0106          |

### Framework Versions
- Python: 3.12.12
- SetFit: 1.1.3
- Sentence Transformers: 5.6.1
- Transformers: 4.57.6
- PyTorch: 2.10.0
- Datasets: 5.0.0
- Tokenizers: 0.22.2

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->