codet5p-portrait / README.md
aalkaswan's picture
Card: query with portraits.query (the build code now lives in one data-portraits repo)
3c86df1 verified
|
Raw History Blame Contribute Delete
2.86 kB
---
tags: [data-portraits, bloom-filter, codet5p, code]
---
# Data Portrait: CodeT5+ training data
A [Data Portrait](https://github.com/ruyimarone/data-portraits) of the pretraining data of [CodeT5+](https://huggingface.co/Salesforce/codet5p-220m) (220M, 770M, their `-py` versions, the 110M
embedding and the 220M bimodal model), rebuilt from
the description in the [CodeT5+ paper](https://arxiv.org/abs/2305.07922) (Section 4.1):
- stage 1, `github-code`: [codeparrot/github-code](https://huggingface.co/datasets/codeparrot/github-code) in Python, Java, Ruby, JavaScript, Go, PHP, C, C++ and C#, permissive licenses only, files of 50 to 2000 whitespace-separated tokens, CodeSearchNet repositories removed
- stage 2, `codesearchnet`: [code-search-net/code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) train split, functions and docstrings
A github-code file's language comes from its path, as in the dataset's loading script. The paper's "50 to 2000
tokens" are read as whitespace-separated tokens: that reproduces its 51.5B CodeT5 tokens (51.6B, extrapolated from 8
of the 1126 files), while counting CodeT5 tokens would give 24.5B. CodeSearchNet repositories (any language or split)
are removed from stage 1 by name, as in the paper.
Which checkpoint saw which stage (model cards): 220M and 770M saw stage 1 only, and their `-py` versions one more
epoch of its Python part; the 110M embedding and 220M bimodal models saw both stages. The 2B, 6B and 16B models were
trained on stage 1 and its Python part too, but they start from CodeGen-mono checkpoints whose training data was
never released.
Superset: the paper also removed repositories that appear in its other evaluation tasks. It does not list them,
so their files are kept here.
Text is normalised as in data-portraits and split into 50-character n-grams with stride
50. Each stage has its own Bloom filter(s) (2 in total), so a match can be traced to
a stage. A query ORs membership across filters: combined false-positive rate <= 0.01 over all of
them (each filter 0.005013).
Each partition is a separate revision of this repo (tag `<stage>-p<i>of<k>`, branch `partition-<n>`):
| stage | tag | files | size |
|---|---|---|---|
| github-code | [`github-code-p0of1`](https://huggingface.co/aalkaswan/codet5p-portrait/tree/github-code-p0of1) | 1126 | 4.0 GiB |
| codesearchnet | [`codesearchnet-p0of1`](https://huggingface.co/aalkaswan/codet5p-portrait/tree/codesearchnet-p0of1) | 6 | 42 MiB |
`partitions.json` lists the exact files (pinned dataset revisions) in each partition. Query with
`portraits.query`:
```python
from portraits.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/codet5p-portrait", ["github-code-p0of1", "codesearchnet-p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]
```