Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Create an atomic structure dataset","local":"create-an-atomic-structure-dataset","sections":[{"title":"Local files","local":"local-files","sections":[],"depth":2},{"title":"PdbFolder and MmcifFolder","local":"pdbfolder-and-mmciffolder","sections":[],"depth":2},{"title":"Cost and preserving source data","local":"cost-and-preserving-source-data","sections":[{"title":"Saving intentional edits","local":"saving-intentional-edits","sections":[],"depth":3},{"title":"Preparing data once for training","local":"preparing-data-once-for-training","sections":[],"depth":3},{"title":"Serialization guarantees","local":"serialization-guarantees","sections":[],"depth":3}],"depth":2},{"title":"Existing datasets and sequence records","local":"existing-datasets-and-sequence-records","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/entry/start.CanNJ44m.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/daWrqvkB.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/3B2wL_Wp.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/entry/app.HcixdYoX.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/BC7CHxXx.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/DXQRzNBd.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/BKhA6VLM.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/nodes/0.BBGy85Wb.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/DkRNWtUN.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/nodes/2.i839-DGI.js" rel="modulepreload"> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/chunks/C67PY0Y1.js" rel="modulepreload"> | |
| <!--149ksp9--><meta name="hf:doc:metadata" content="{"title":"Create an atomic structure dataset","local":"create-an-atomic-structure-dataset","sections":[{"title":"Local files","local":"local-files","sections":[],"depth":2},{"title":"PdbFolder and MmcifFolder","local":"pdbfolder-and-mmciffolder","sections":[],"depth":2},{"title":"Cost and preserving source data","local":"cost-and-preserving-source-data","sections":[{"title":"Saving intentional edits","local":"saving-intentional-edits","sections":[],"depth":3},{"title":"Preparing data once for training","local":"preparing-data-once-for-training","sections":[],"depth":3},{"title":"Serialization guarantees","local":"serialization-guarantees","sections":[],"depth":3}],"depth":2},{"title":"Existing datasets and sequence records","local":"existing-datasets-and-sequence-records","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/datasets/pr_8895/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: inline-end; margin-inline-start: 10px; display: inline-flex; position: relative; z-index: 10;"><div dir="ltr" class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----></span> <span dir="auto">Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg><!----></button></div> <!--[-1--><!--]--></div><!----> <!--[0--><h1 class="relative group"><a id="create-an-atomic-structure-dataset" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#create-an-atomic-structure-dataset"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Create an atomic structure dataset<!----></span></h1><!--]--><!----> <p>There are two methods for creating and sharing an atomic structure dataset. This guide will show you how to:</p> <ul><li><p>Create an atomic structure dataset from local files in python with <a href="/docs/datasets/pr_8895/en/package_reference/main_classes#datasets.Dataset.push_to_hub">Dataset.push_to_hub()</a>. This is an easy way that requires only a few steps in python.</p></li> <li><p>Create an atomic structure dataset with <code>PdbFolder</code> or <code>MmcifFolder</code> and some metadata. This is a no-code solution for quickly creating a dataset with several thousand structures.</p></li></ul> <p>Source paths and bytes are stored unchanged. Structures are read with AtomWorks only when a row is decoded; casting a column does not parse its files. Live atom arrays, RDKit molecules and ASE objects are serialized as BinaryCIF.</p> <p>AtomicStructure decoding requires <a href="https://github.com/RosettaCommons/atomworks" rel="nofollow">AtomWorks 3.1.0 or newer</a> and Python 3.12 or newer:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-sh "><!---->pip install <span class="hljs-string">"datasets[bio]"</span><!----></pre></div><!----> <p>The <code>bio</code> extra installs base AtomWorks on Python 3.12+ and Biopython for the legacy features. AtomWorks’ <code>ml</code>, <code>hf</code>, <code>cli</code> and <code>dev</code> extras are not needed; install <code>ase</code> separately only for ASE inputs. Let AtomWorks select its compatible Biotite version. On Python 3.10/3.11 the legacy features still work, and storing file references or using <code>AtomicStructure(decode=False)</code> does not need AtomWorks.</p> <blockquote class="tip"><p>You can control access to your dataset by requiring users to share their contact information first. Check out the <a href="https://huggingface.co/docs/hub/datasets-gated" rel="nofollow">Gated datasets</a> guide for more information about how to enable this feature on the Hub.</p></blockquote> <!--[1--><h2 class="relative group"><a id="local-files" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#local-files"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Local files<!----></span></h2><!--]--><!----> <p>You can load your own dataset using the paths to your structure files. Use the <a href="/docs/datasets/pr_8895/en/package_reference/main_classes#datasets.Dataset.cast_column">cast_column()</a> function to take a column of file paths, and cast it to the <a href="/docs/datasets/pr_8895/en/package_reference/main_classes#datasets.AtomicStructure">AtomicStructure</a> feature:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> Dataset, AtomicStructure | |
| <span class="hljs-meta">>>> </span>structure_dataset = Dataset.from_dict({<span class="hljs-string">"structure"</span>: [<span class="hljs-string">"path/to/1abc.cif.gz"</span>, <span class="hljs-string">"path/to/ligand.sdf"</span>, <span class="hljs-string">"path/to/2xyz.pdb"</span>]}).cast_column(<span class="hljs-string">"structure"</span>, AtomicStructure()) | |
| <span class="hljs-meta">>>> </span>structure_dataset[<span class="hljs-number">0</span>][<span class="hljs-string">"structure"</span>] | |
| <atomworks.io.utils.atom_array_plus.AtomArrayPlus ...><!----></pre></div><!----> <p>Decoded output is an <code>AtomArrayPlus</code> for one model, an <code>AtomArrayPlusStack</code> for multiple models with matching atoms, or a list of arrays for models with differing atom sets.</p> <p>mmCIF (<code>.cif</code>, <code>.mmcif</code>), BinaryCIF (<code>.bcif</code>), PDB (<code>.pdb</code>, <code>.ent</code>), MOL/SDF (<code>.mol</code>, <code>.sdf</code>) and Mol2 (<code>.mol2</code>) files are supported, optionally compressed with gzip, bz2 or zstd. RDKit <code>Mol</code>, ASE <code>Atoms</code> and biotite <code>AtomArray</code> objects can be used instead of paths.</p> <p>By default decoding uses AtomWorks’ <code>minimal</code> preset: waters, hydrogens and crystallization aids are kept, and no missing atoms or biological assemblies are added. Minimal parsing still selects the first alternate conformer, normalizes identifiers and infers bonds. This is a parsed representation, not an exact copy of every file field; the stored source bytes remain unchanged. Use <code>decode=False</code> to retrieve the original encoding.</p> <p>Pass <code>parse_config</code> to opt into preparation at <strong>decode time</strong>. Unspecified options use the minimal preset; for example <code>AtomicStructure(parse_config={"remove_waters": True})</code>. Presets such as <code>default</code> or <code>rcsb</code> opt into their full cleanup, including atom completion and filtering. <code>assembly</code> opts into building a biological assembly. These settings also apply to BinaryCIF inputs and can be changed with <code>cast_column</code> without rewriting stored data:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-meta">>>> </span>feature = AtomicStructure(parse_config={<span class="hljs-string">"preset"</span>: <span class="hljs-string">"rcsb"</span>}, assembly=<span class="hljs-string">"1"</span>) | |
| <span class="hljs-meta">>>> </span>structure_dataset = Dataset.from_dict({<span class="hljs-string">"structure"</span>: [<span class="hljs-string">"path/to/1abc.cif.gz"</span>]}).cast_column(<span class="hljs-string">"structure"</span>, feature)<!----></pre></div><!----> <p>Then upload the dataset to the Hugging Face Hub using <a href="/docs/datasets/pr_8895/en/package_reference/main_classes#datasets.Dataset.push_to_hub">Dataset.push_to_hub()</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!---->structure_dataset.push_to_hub(<span class="hljs-string">"<username>/my_dataset"</span>)<!----></pre></div><!----> <p>This will create a dataset repository containing your atomic structure dataset:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-text "><!---->my_dataset/README.md | |
| my_dataset/data/train-00000-of-00001.parquet<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="pdbfolder-and-mmciffolder" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#pdbfolder-and-mmciffolder"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->PdbFolder and MmcifFolder<!----></span></h2><!--]--><!----> <p><code>PdbFolder</code> and <code>MmcifFolder</code> are dataset builders designed to quickly load a dataset with several thousand structure files without requiring you to write any code. <code>PdbFolder</code> reads <code>.pdb</code> and <code>.ent</code> files, <code>MmcifFolder</code> reads <code>.cif</code>, <code>.mmcif</code> and <code>.bcif</code> files.</p> <blockquote class="tip"><p>Take a look at the <a href="repository_structure#split-pattern-hierarchy">Split pattern hierarchy</a> to learn more about how these builders create dataset splits based on your dataset repository structure.</p></blockquote> <p>They automatically infer the class labels of your dataset based on the directory name. Store your dataset in a directory structure like:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-text "><!---->folder/train/enzyme/1abc.pdb | |
| folder/train/enzyme/2def.pdb | |
| folder/train/receptor/3ghi.pdb | |
| folder/train/receptor/4jkl.pdb<!----></pre></div><!----> <p>If the dataset follows the folder structure, then you can load it directly with <a href="/docs/datasets/pr_8895/en/package_reference/loading_methods#datasets.load_dataset">load_dataset()</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| <span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"path/to/folder"</span>)<!----></pre></div><!----> <p>This is equivalent to passing <code>pdb</code> manually in <a href="/docs/datasets/pr_8895/en/package_reference/loading_methods#datasets.load_dataset">load_dataset()</a> and the directory in <code>data_dir</code>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-meta">>>> </span>dataset = load_dataset(<span class="hljs-string">"pdb"</span>, data_dir=<span class="hljs-string">"/path/to/folder"</span>)<!----></pre></div><!----> <p>If there is additional information you’d like to include about your dataset, like resolution or experimental method, add it as a <code>metadata.csv</code> file in your folder. You can also use a JSONL file <code>metadata.jsonl</code> or a Parquet file <code>metadata.parquet</code>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-text "><!---->folder/train/metadata.csv | |
| folder/train/1abc.pdb | |
| folder/train/2def.pdb<!----></pre></div><!----> <p>Your <code>metadata.csv</code> file must have a <code>file_name</code> or <code>*_file_name</code> field which links structure files with their metadata:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-csv "><!----><span class="hljs-attribute">file_name</span>,resolution | |
| <span class="hljs-attribute">1abc</span>.pdb,<span class="hljs-number">2</span>.<span class="hljs-number">5</span> | |
| <span class="hljs-attribute">2def</span>.pdb,<span class="hljs-number">1</span>.<span class="hljs-number">8</span><!----></pre></div><!----> <p>Here the <code>file_name</code> must be the name of the structure file next to the metadata file. More generally, it must be the relative path from the directory containing the metadata to the structure file.</p> <p>Finally, <a href="/docs/datasets/pr_8895/en/package_reference/main_classes#datasets.Dataset.push_to_hub">push_to_hub()</a> the dataset to the Hub:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-meta">>>> </span>dataset.push_to_hub(<span class="hljs-string">"<username>/my_dataset"</span>)<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="cost-and-preserving-source-data" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#cost-and-preserving-source-data"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Cost and preserving source data<!----></span></h2><!--]--><!----> <p>Creating or casting a dataset of paths does no structure I/O. Byte inputs (including compressed files) are retained exactly. Saving or uploading embeds files without chemical preparation. Decoding pays the read cost for rows accessed; opt-in preparation adds its own per-read cost. AtomWorks’ first import and CCD initialization may be expensive in each worker. Repeated reads repeat the work; for frequently reused prepared data, explicitly materialize it with <code>Dataset.map</code> after choosing your scientific preprocessing policy.</p> <!--[2--><h3 class="relative group"><a id="saving-intentional-edits" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#saving-intentional-edits"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Saving intentional edits<!----></span></h3><!--]--><!----> <p>Decoded arrays retain their original encoding in the <code>_hf_encoded</code> system annotation (<code>atoms.get_annotation("_hf_encoded", n_body=0)</code>), so a pass-through <code>map</code> preserves source bytes and metadata. To save edits explicitly, use <code>materialize</code>. It serializes the atoms as supplied, ignores the retained bytes, leaves the input unchanged, and does no additional preparation:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> AtomicStructure, Features | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">def</span> <span class="hljs-title function_">translate</span>(<span class="hljs-params">example</span>): | |
| <span class="hljs-meta">... </span> atoms = example[<span class="hljs-string">"structure"</span>] | |
| <span class="hljs-meta">... </span> atoms.coord += [<span class="hljs-number">1.0</span>, <span class="hljs-number">0.0</span>, <span class="hljs-number">0.0</span>] | |
| <span class="hljs-meta">... </span> <span class="hljs-keyword">return</span> {<span class="hljs-string">"structure"</span>: AtomicStructure().materialize(atoms)} | |
| <span class="hljs-meta">>>> </span>edited = structure_dataset.<span class="hljs-built_in">map</span>( | |
| <span class="hljs-meta">... </span> translate, | |
| <span class="hljs-meta">... </span> features=Features({**structure_dataset.features, <span class="hljs-string">"structure"</span>: AtomicStructure()}), | |
| <span class="hljs-meta">... </span>)<!----></pre></div><!----> <p>The plain output feature avoids applying a previous cleanup policy or assembly expansion a second time. <code>materialize</code> also accepts stacks and nonempty lists of models with different atoms. The original dataset remains available; choose a separate output column or save location when both representations are needed.</p> <!--[2--><h3 class="relative group"><a id="preparing-data-once-for-training" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#preparing-data-once-for-training"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Preparing data once for training<!----></span></h3><!--]--><!----> <p>Cleanup and assembly construction can be expensive. Apply them once with <code>map</code>, explicitly materialize the result, then save it for training. The function below is ordinary Python so you can add your own filters or transformations:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-meta">>>> </span><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> AtomicStructure, Features | |
| <span class="hljs-meta">>>> </span>policy = AtomicStructure(parse_config={<span class="hljs-string">"remove_waters"</span>: <span class="hljs-literal">True</span>, <span class="hljs-string">"altloc"</span>: <span class="hljs-string">"first"</span>}) | |
| <span class="hljs-meta">>>> </span>source = structure_dataset.cast_column(<span class="hljs-string">"structure"</span>, policy) | |
| <span class="hljs-meta">>>> </span><span class="hljs-keyword">def</span> <span class="hljs-title function_">prepare</span>(<span class="hljs-params">example</span>): | |
| <span class="hljs-meta">... </span> atoms = example[<span class="hljs-string">"structure"</span>] <span class="hljs-comment"># decoding applies policy here</span> | |
| <span class="hljs-meta">... </span> <span class="hljs-keyword">return</span> {<span class="hljs-string">"structure"</span>: <span class="hljs-literal">None</span> <span class="hljs-keyword">if</span> atoms <span class="hljs-keyword">is</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">else</span> AtomicStructure().materialize(atoms)} | |
| <span class="hljs-meta">>>> </span>prepared = source.<span class="hljs-built_in">map</span>( | |
| <span class="hljs-meta">... </span> prepare, | |
| <span class="hljs-meta">... </span> features=Features({**source.features, <span class="hljs-string">"structure"</span>: AtomicStructure()}), | |
| <span class="hljs-meta">... </span> load_from_cache_file=<span class="hljs-literal">False</span>, | |
| <span class="hljs-meta">... </span>) | |
| <span class="hljs-meta">>>> </span>prepared.save_to_disk(<span class="hljs-string">"training-structures"</span>)<!----></pre></div><!----> <p>The plain output feature prevents cleanup or assembly expansion from running again. Reading the saved dataset still parses BinaryCIF. Other columns and null rows are retained, and the source dataset is unchanged. Start with one worker before adding <code>num_proc</code>: each process initializes AtomWorks separately.</p> <p>This example disables reuse of <code>map</code> caches because external files and CCD definitions can change without changing a dataset fingerprint. Reuse the saved dataset only for the same source snapshot, preparation settings and AtomWorks version. If you enable caching in a custom pipeline, include source content hashes, settings, dependency versions and external CCD versions in its cache identity.</p> <!--[2--><h3 class="relative group"><a id="serialization-guarantees" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#serialization-guarantees"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Serialization guarantees<!----></span></h3><!--]--><!----> <table><thead><tr><th>Input or property</th><th>Guarantee</th></tr></thead><tbody><tr><td>Source file bytes, including compression</td><td>Preserved exactly until explicit materialization</td></tr><tr><td>Materialized coordinates</td><td>No deliberate coordinate rounding; float32 precision tested</td></tr><tr><td>Bonds</td><td>Standard CIF connectivity and AtomWorks inference; no exact edited-graph preservation</td></tr><tr><td>Periodic boxes</td><td>No round-trip guarantee</td></tr><tr><td>Scalar per-atom annotations</td><td>Values tested for integers, floats, booleans and strings; inferred dtypes may change</td></tr><tr><td>Registered StandardAnnotations</td><td>Supported by the upstream writer; regression-tested</td></tr><tr><td>Arbitrary custom multidimensional, system-wide or pairwise annotations</td><td>Rejected explicitly</td></tr><tr><td>Integer annotations outside signed 32-bit range</td><td>Rejected explicitly</td></tr><tr><td>Empty models</td><td>Rejected by materialization; filter these rows before preparing storage</td></tr></tbody></table> <p>CIF serialization can normalize author identifiers and annotation dtypes. It does not preserve all original file categories or custom Python objects; retain the original source dataset when that information matters. Bonds are read from standard CIF categories and inferred by AtomWorks; edited bond graphs, missing-versus-empty bonds and periodic boxes are not guaranteed to survive materialization. Keep the source bytes when exact file preservation matters. These are format-specific guarantees, not a general arbitrary-array codec.</p> <!--[1--><h2 class="relative group"><a id="existing-datasets-and-sequence-records" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#existing-datasets-and-sequence-records"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Existing datasets and sequence records<!----></span></h2><!--]--><!----> <p><code>BioStructure</code> and <code>BioSequence</code> remain importable and registered, including their original serialized feature names and Biopython output types. Saved datasets continue to load. For new structure datasets prefer <code>AtomicStructure</code>; migrate an existing column explicitly with <code>cast_column("structure", AtomicStructure())</code>.</p> <p>FASTA, FASTQ and GenBank keep their existing <code>record</code> columns and schemas. In particular, GenBank records retain sections such as REFERENCE and COMMENT that are absent from parsed columns. To avoid duplicate FASTA storage, choose <code>columns=["id", "description", "sequence"]</code> when loading. Changing sequence-loader defaults is separate from adopting AtomicStructure.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/datasets/blob/main/docs/source/atomic_structure_dataset.mdx" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_1kv630f = { | |
| base: "/docs/datasets/pr_8895/en", | |
| assets: "/docs/datasets/pr_8895/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/datasets/pr_8895/en/_app/immutable/entry/start.CanNJ44m.js"), | |
| import("/docs/datasets/pr_8895/en/_app/immutable/entry/app.HcixdYoX.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 2], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 49 kB
- Xet hash:
- bd4389d5020847b299f9594a75f98787da4864a480d6a8e207608a1af312a69b
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.