From Genomic Sequences to Taxonomic IDs (Part 1): Building a Scalable Enrichment Pipeline for the Carbon Pretraining Corpus
TL;DR Raw genomic sequences hold biological signal that's hard to read without context. We built a CPU-only pipeline that enriches the carbon-pretraining-corpus with sequence length, GC content, coding status, and taxonomy with no Carbon model involved. The output: a 32.4M-record enriched corpus, providing a structured foundation for biological analysis.
Why Enrich a Genomic Pretraining Corpus?
A genomic pretraining corpus is designed primarily for foundational model training. It contains raw sequences corresponding to a given taxonomic record and contains additional attributes such as gene_type, begin_of_sequence, end_of_sequence, etc. There is no per-record answer to basic biological questions such as what is its nucleotide composition, how does the carbon model score the sequence or how long is the sequence itself.
Closing the gap between biological context and raw sequence data requires enriching the data with structured biological information to a sequence, at a scale that matches the corpus itself. That is the purpose of the enrichment this series describes.
Defining Enrichment in Practice
Enrichment can be defined in a specific manner. We take a raw sequence record and attach computed metrics as structured fields. It happens in two layers and is performed to the eukaryote_generator split of the carbon-pretraining-corpus.
Figure 1. Composition of an Enriched Record
The first is CPU side: Properties computable directly from the sequence string and the accompanying metadata fields. They are calculated as an inexpensive operation and a greater sample of the corpus. The second layer, which would be covered in a future article is GPU side: Representations that only Carbon-3B itself can produce such as likelihood scores and embeddings. This is expensive per record and runs over a sampled cohort. This split is the central architectural decision and shapes every major design choice behind the pipeline.
Together, these 2 layers transform the pretraining corpus into a structured set of resources for structured analysis. It preserves a clear distinction between properties that can be computed directly from a sequence and information that emerges through a pretrained biological model, providing the basis for the analysis that follows.
Establishing the Execution and Asset Model
A data enrichment pipeline requires more than assets that can transform records. As the number of processing stages increases, the system also needs consistent access to models, dataset clients, analytical engines and catalog metadata. Without this separation, infrastructure becomes coupled to individual assets and downstream analytics.
View the Interactive Execution Diagram
The raw corpus contains molecule type, topology, taxonomy and sequence boundaries. Instead of distributing field definitions and output expectations across individual processing components, we define a single source of truth as the schema layer. A key responsibility of the layer is defining biological identity. The layer defines the expected data types for these fields. In particular, taxonomy remains a variable-length field rather than being truncated into a fixed-width structure. The faceberg resource maps existing Hugging Face datasets to Apache Iceberg metadata without copying the underlying data. The analytical engines serve different access patterns. duckdb and ClickHouse support structured analytical queries, while lancedb provides vector search and access to embedding data.
Furthermore, it declares the expected output schemas for tokenized sequences, embeddings and likelihood statistics. The record_id field represents the NCBI ID, and a single record_id can contain multiple sequence intervals. Therefore, the pipeline represents the identity of each interval using the composite key (record_id, start, end).
Building the Foundation for Configurable Enrichment
The resource layer provides the capabilities that pipeline components consume from analytical operations to catalog access. They support downstream querying, lineage management and vector analysis without requiring every operation to be expressed as an orchestrated asset. Next, we define a configuration layer to apply a clear boundary for controlling pipeline behaviour with YAML-based run configurations, allowing execution behavior to be adjusted without modifying asset implementations. Configuration controls how a pipeline run is parameterized, while the orchestration is controlled by asset definitions to describe what the pipeline materializes.
The separation between configuration and orchestration also establishes a clearer division of responsibility while the schema establishes the data contract for the pretraining corpus, defining what each record contains and which structural conditions must remain consistent across processing stages. With these boundaries established, the next challenge is to process and materialize the large corpus efficiently while preserving the data contract throughout execution.
Establishing a Data Catalog for Genomic Enrichment
As the scope of the enrichment pipeline increased, there were several intermediate outputs and artifacts such as tokenized sequences, sampled subsets, embeddings, etc. Each artifact represents a different stage of the same enrichment process, but the relationships between them could not be reliably inferred from the files alone.
Hence, we add a catalog layer based on faceberg and duckdb, with faceberg providing the catalog and duckdb being the execution interface for provenance. The purpose of this layer is to provide a structured metadata layer describing the datasets, their schemas, locations, lineage and provenance.
pretraining_corpus:
table: null
repo: HuggingFaceBio/carbon-pretraining-corpus
config: null
upstream: null
access_mode: streaming
description: Root source corpus (eukaryote_generator/train split).
cpu_enriched:
table: carbon.cpu_enriched_sequences
repo: AINovice2005/carbon-cpu-enriched-sequences
config: null
upstream: pretraining_corpus
access_mode: catalog
description: CPU-derived sequence features from the 75% pretraining split.
sampled_cpu:
table: carbon.pilot_corpus_dedup
repo: AINovice2005/carbon-cpu-enriched-sequences-sampled
config: null
upstream: cpu_enriched
access_mode: catalog
description: Stratified CPU-enriched population used as the GPU input corpus.
tokenized:
table: carbon.tokenized_corpus
repo: AINovice2005/carbon-tokenized-corpus
config: null
upstream: sampled_cpu
access_mode: catalog
description: Model-ready tokenized input from the sampled CPU population.
likelihood_stats:
table: carbon.likelihood_stats
repo: AINovice2005/carbon-likelihood-stats
config: null
upstream: tokenized
access_mode: catalog
description: Per-sequence model likelihood statistics from GPU enrichment.
embeddings:
table: carbon.embeddings
repo: AINovice2005/carbon-embeddings
config: null
upstream: tokenized
access_mode: catalog
description: Model-derived sequence embeddings from GPU enrichment.
Figure 3: Dataset Lineage (lineage.yml) Across the Data Enrichment Pipeline
The resulting metadata is stored with the faceberg configuration, lineage information and metadata in the carbon-catalog bucket, alongside the repo. It establishes a relationship between the artifacts produced by successive stages. Furthermore, we can also describe what a dataset represents, how it was produced, which schema it follows and where it belongs in the enrichment lineage. This allows for a system in which biological enrichment and its provenance can be inspected together rather than being viewed as disconnected artifacts.
Designing the CPU Pipeline
The CPU enrichment pipeline is built as a streaming, memory-bounded dagster asset, with dagster and dagster-hf-datasets forming the orchestration layer for HF Datasets. The pipeline treats the source dataset as a stream of records and processes it through a sequence of transformations. The streaming dataset pipeline makes CPU-based enrichment applicable for a greater set of records without introducing the computational cost of in-memory operations.
View the Interactive CPU Pipeline Diagram
Keeping the corpus in motion with Streaming and Multiprocessing
The pipeline begins with create_carbon_stream() which opens the dataset split with streaming access and applies a configured limit through IterableDataset.take() . The records are passed to iter_batches(), which groups them into pyarrow.RecordBatch objects. This method ensures that PyArrow is applied as the common transport format between ingestion and processing, reducing the need for repeated conversion into Python based lists or other intermediate data structures.
Furthermore, we apply parallel execution through asynchronous calculation of results. The pipeline uses Python’s ProcessPoolExecutor to distribute batch-level enrichment across multiple worker processes. The worker subsection performs 3 complementary operations: enrichment, validation and normalization. Validation maintains worker-local ValidationStats instances to track checks such as sequence-length ranges, taxonomy depth, token violations and boundary mismatches. Normalization uses vectorized PyArrow operations to standardize sequence strings and taxonomy fields while preserving the existing column structure.
Completed workers are consumed using as_completed() , allowing the parent process to finish results as they become available. The number of in-flight tasks is limited to n_workers * 3 by the pipeline to prevent deadlocks and avoid pending batches in memory. Validation statistics are maintained locally during computation and merged by the parent process as completed results are collected. As enrichment is completed, the completed batches are written incrementally through ParquetShardWriter . The writer persists records as Zstandard compressed Parquet files and rotates to a new shard for a configurable limit. This allows to achieve a high throughput of enrichment while avoid retaining the results as an intermediate data type at write time.
The Dagster asset pipeline also includes observability about details such as number of rows read, batches processed, batch size, shards written, etc. This allows to monitor the enrichment process for issues and allows to address 3 core concerns within one processing architecture: Efficient Sequence Transformation, Incremental Persistence and Execution Observability
The resulting architecture establishes the CPU-enriched dataset as a set of analyzable records and preserves the distinction between source metadata and computed features.
Comprehending the resulting dataset looks like
The resulting dataset contains 22 features, 13 original Carbon fields and 9 derived enriched features. The schema therefore contains both record-level metadata and biological descriptors computed directly from the underlying nucleotide sequence.
The original fields preserve the identity, genomic context, sequence representation, strand, molecular properties, and taxonomic lineage of each record. The enrichment features extend this representation with sequence-level measurements such as GC content, GC skew, sequence complexity, 3-mer frequency profiles, normalized strand representation and taxonomic depth. We can now analyze the resulting feature schema in the next sections.
Mapping Taxonomic Coverage Across the Enriched Corpus
Figure 5: Distinct taxa represented across Hierarchical Ranks
The bar chart shows the breadth of taxonomic information represented in the CPU-enriched corpus. Taxonomic diversity increases across the displayed hierarchy, with a relatively small number of distinct labels at higher ranks and substantially more distinct labels at finer ranks. The presence of 1,294 distinct genera, 1,106 distinct suborders and 1,186 distinct families indicate that the corpus contains detailed taxonomic annotations across a wide range of lower-level classifications.
This distribution illustrates the taxonomic granularity of the enriched dataset. It shows that the corpus can be organized and analyzed at different levels of biological classification, supporting comparisons between broad taxonomic groups and more specific lineages.
Figure 6: Broad sequence-length and GC-content coverage
In the second figure, the heatmap explores the joint distribution of sequence length and GC content in across the enriched genomic corpus. Each cell represents the number of sequences falling into a particular combination of length and GC-content buckets, with color intensity corresponding to the logarithmically transformed sequence count, log10(count + 1).
The corpus is not uniformly distributed across the feature space. Instead, sequences cluster around particular combinations of length and GC content, while other combinations occur less frequently. The highest-density region is centered approximately around sequence-length buckets 15–25 and GC-content buckets 14–25, indicating that a substantial portion of the corpus occupies this region of the two-dimensional feature space.
These concentrations can be examined further to determine whether they reflect specific biological groups, source-record characteristics, or preprocessing effects.
Characterizing Sequence Composition Through Length and GC-Content Distributions
Figure 7: GC-content distribution of the corpus
The GC-content distribution provides an overview of the nucleotide composition of the 32.41 million sequences in the CPU-enriched corpus. The histogram divides sequences into GC-content intervals ranging from 0–5% to 95–100%, allowing us to examine how the corpus is distributed across different levels of guanine and cytosine content.
The distribution is concentrated primarily in the 35–55% GC-content range. The 35–40% bucket contains the largest number of sequences, with 7,554,308 records, followed by the 40–45% bucket with 5,569,451 records. The 50–55% bucket contains 4,804,255 sequences, while the 45–50% bucket contains 4,427,297 sequences. The lower GC-content intervals also contain relatively few sequences. This figure provides a reference for examining whether model-derived features vary across sequences with different GC-content characteristics.
Figure 8: Sequence-length distribution of the Corpus
The sequence-length distribution describes the size of the genomic records represented in the CPU-enriched corpus. The histogram groups sequences into length intervals ranging from records shorter than 50 bases to sequences between 500 and 999 bases, providing a view of the size composition of the dataset.
The distribution is dominated by shorter genomic sequences. The corpus also includes a substantial number of sequences in the 500–999 base interval, which contains 4,056,019 records. The number of sequences decreases across several of the longer length intervals. The 25K–50K bucket contains 1,430,764 sequences, while the 50K+ bucket contains 915,686. The shortest category, below 50 bases, contains 18,653 sequences, indicating limited representation within that specific length interval.
The distribution shows that the enriched corpus contains a mixture of sequence sizes, with the largest concentration in the shorter and intermediate length ranges represented by the histogram. This provides an initial understanding of the size characteristics of the corpus and establishes a reference for subsequent sequence-level and model-based analyses. In the next section, we take a granular look at the enrichment as well.
Connecting Enriched Sequences to Biological Context Through Interactive Record-Level Analysis
Sequence-level enrichment provides a structured representation of genomic intervals, but aggregate statistics alone are insufficient to understand the biological variation within a targeted cohort. To bridge this gap, we developed a taxonomy-targeted Marimo notebook that connects measurable sequence properties with record-level exploration. The demonstrated analysis focuses on the fungal taxonomic lineage Fungi → Pleosporales → Pleosporaceae → Alternaria,.
The notebook begins by selecting the Alternaria taxonomic target and extracting the corresponding intervals from the CPU-enriched dataset. Rather than materializing the full dataset in Python, the ingestion stage pushes taxonomy filtering into ClickHouse and writes matching intervals to a local Parquet cache. Subsequent analyses operate on this cached cohort, allowing the same selected population to be examined across multiple visualizations.
The resulting views progress from basic sequence composition to record-level heterogeneity. A record-level biological profile heatmap then aggregates interval-level measurements by accession, providing a compact view of differences in sequence length, GC content and interval count.
Figure 9: Record Level Biological Heatmap
The CPU enrichment pipeline provides a record-level view of biological variation through an interactive heatmap, shown in Figure 9. To make the features comparable, each column of the heatmap represents a record-level aggregate, while the feature values are standardized as z-scores across the displayed records. The resulting color intensity indicates how far a record's feature value deviates from the corresponding feature mean.
The heatmap reveals substantial variation in the number of intervals associated with individual records, with a small number of records exhibiting elevated interval counts relative to the displayed cohort. Coding proportion remains comparatively consistent across most records, although isolated deviations are visible. Notably, some records exhibit simultaneous deviations across multiple features, providing candidates for further investigation into their underlying sequence and taxonomic characteristics.
What’s next?
In this manner, we establish the foundation for analyzing the carbon-pretraining-corpus as a structured dataset. By augmenting raw sequences with derived statistics and nucleotide-based information, we transform a static set of records into an auditable dataset that can be examined across biological records. In Part 2, we shall focus on the GPU enrichment pipeline, its architecture and generation of model-derived features. The enrichment presented here help to connect raw genomic sequences to biological context, which enables us to keenly analyze the information inside the pretraining corpus.
Resources
Primary Dataset — The 32.4M-record CPU-enriched corpus containing sequence-level biological features, taxonomy, metadata and quality-control fields.
Code Repo — The implementation of the enrichment pipeline, including the Dagster asset graph,configuration and pipeline documentation.
Dataset Collection — The complete collection of enriched datasets produced by the pipeline, including CPU-derived sequence features, model likelihood statistics, embeddings and intermediate tokenized datasets.
