|
Download README.md from aalkaswan/codet5p-portrait: direct link, hf CLI and curl.
- Browser
- Download file 2.86 kB
-
https://huggingface.co/aalkaswan/codet5p-portrait/resolve/main/README.md
- Command line
-
hf download hf://aalkaswan/codet5p-portrait/README.md
-
curl -L -o README.md https://huggingface.co/aalkaswan/codet5p-portrait/resolve/main/README.md
2.86 kB
| tags: [data-portraits, bloom-filter, codet5p, code] | |
| # Data Portrait: CodeT5+ training data | |
| A [Data Portrait](https://github.com/ruyimarone/data-portraits) of the pretraining data of [CodeT5+](https://huggingface.co/Salesforce/codet5p-220m) (220M, 770M, their `-py` versions, the 110M | |
| embedding and the 220M bimodal model), rebuilt from | |
| the description in the [CodeT5+ paper](https://arxiv.org/abs/2305.07922) (Section 4.1): | |
| - stage 1, `github-code`: [codeparrot/github-code](https://huggingface.co/datasets/codeparrot/github-code) in Python, Java, Ruby, JavaScript, Go, PHP, C, C++ and C#, permissive licenses only, files of 50 to 2000 whitespace-separated tokens, CodeSearchNet repositories removed | |
| - stage 2, `codesearchnet`: [code-search-net/code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) train split, functions and docstrings | |
| A github-code file's language comes from its path, as in the dataset's loading script. The paper's "50 to 2000 | |
| tokens" are read as whitespace-separated tokens: that reproduces its 51.5B CodeT5 tokens (51.6B, extrapolated from 8 | |
| of the 1126 files), while counting CodeT5 tokens would give 24.5B. CodeSearchNet repositories (any language or split) | |
| are removed from stage 1 by name, as in the paper. | |
| Which checkpoint saw which stage (model cards): 220M and 770M saw stage 1 only, and their `-py` versions one more | |
| epoch of its Python part; the 110M embedding and 220M bimodal models saw both stages. The 2B, 6B and 16B models were | |
| trained on stage 1 and its Python part too, but they start from CodeGen-mono checkpoints whose training data was | |
| never released. | |
| Superset: the paper also removed repositories that appear in its other evaluation tasks. It does not list them, | |
| so their files are kept here. | |
| Text is normalised as in data-portraits and split into 50-character n-grams with stride | |
| 50. Each stage has its own Bloom filter(s) (2 in total), so a match can be traced to | |
| a stage. A query ORs membership across filters: combined false-positive rate <= 0.01 over all of | |
| them (each filter 0.005013). | |
| Each partition is a separate revision of this repo (tag `<stage>-p<i>of<k>`, branch `partition-<n>`): | |
| | stage | tag | files | size | | |
| |---|---|---|---| | |
| | github-code | [`github-code-p0of1`](https://huggingface.co/aalkaswan/codet5p-portrait/tree/github-code-p0of1) | 1126 | 4.0 GiB | | |
| | codesearchnet | [`codesearchnet-p0of1`](https://huggingface.co/aalkaswan/codet5p-portrait/tree/codesearchnet-p0of1) | 6 | 42 MiB | | |
| `partitions.json` lists the exact files (pinned dataset revisions) in each partition. Query with | |
| `portraits.query`: | |
| ```python | |
| from portraits.query import SequentialPortrait | |
| portrait = SequentialPortrait("aalkaswan/codet5p-portrait", ["github-code-p0of1", "codesearchnet-p0of1"]) | |
| report = portrait.contains_from_text([text], sort_chains_by_length=True)[0] | |
| ``` | |