--- tags: [data-portraits, bloom-filter, codet5p, code] --- # Data Portrait: CodeT5+ training data A [Data Portrait](https://github.com/ruyimarone/data-portraits) of the pretraining data of [CodeT5+](https://huggingface.co/Salesforce/codet5p-220m) (220M, 770M, their `-py` versions, the 110M embedding and the 220M bimodal model), rebuilt from the description in the [CodeT5+ paper](https://arxiv.org/abs/2305.07922) (Section 4.1): - stage 1, `github-code`: [codeparrot/github-code](https://huggingface.co/datasets/codeparrot/github-code) in Python, Java, Ruby, JavaScript, Go, PHP, C, C++ and C#, permissive licenses only, files of 50 to 2000 whitespace-separated tokens, CodeSearchNet repositories removed - stage 2, `codesearchnet`: [code-search-net/code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) train split, functions and docstrings A github-code file's language comes from its path, as in the dataset's loading script. The paper's "50 to 2000 tokens" are read as whitespace-separated tokens: that reproduces its 51.5B CodeT5 tokens (51.6B, extrapolated from 8 of the 1126 files), while counting CodeT5 tokens would give 24.5B. CodeSearchNet repositories (any language or split) are removed from stage 1 by name, as in the paper. Which checkpoint saw which stage (model cards): 220M and 770M saw stage 1 only, and their `-py` versions one more epoch of its Python part; the 110M embedding and 220M bimodal models saw both stages. The 2B, 6B and 16B models were trained on stage 1 and its Python part too, but they start from CodeGen-mono checkpoints whose training data was never released. Superset: the paper also removed repositories that appear in its other evaluation tasks. It does not list them, so their files are kept here. Text is normalised as in data-portraits and split into 50-character n-grams with stride 50. Each stage has its own Bloom filter(s) (2 in total), so a match can be traced to a stage. A query ORs membership across filters: combined false-positive rate <= 0.01 over all of them (each filter 0.005013). Each partition is a separate revision of this repo (tag `-pof`, branch `partition-`): | stage | tag | files | size | |---|---|---|---| | github-code | [`github-code-p0of1`](https://huggingface.co/aalkaswan/codet5p-portrait/tree/github-code-p0of1) | 1126 | 4.0 GiB | | codesearchnet | [`codesearchnet-p0of1`](https://huggingface.co/aalkaswan/codet5p-portrait/tree/codesearchnet-p0of1) | 6 | 42 MiB | `partitions.json` lists the exact files (pinned dataset revisions) in each partition. Query with `portraits.query`: ```python from portraits.query import SequentialPortrait portrait = SequentialPortrait("aalkaswan/codet5p-portrait", ["github-code-p0of1", "codesearchnet-p0of1"]) report = portrait.contains_from_text([text], sort_chains_by_length=True)[0] ```