ColiFormer Training and Evaluation Dataset

This dataset contains the training and evaluation data used for the ColiFormer model - a specialized codon optimization transformer fine-tuned for Escherichia coli sequences. The model achieves 6.2% better CAI (Codon Adaptation Index) scores compared to the base CodonTransformer model.

🔗 Related Resources

• Model: saketh11/ColiFormer
• Base Model: adibvafa/CodonTransformer
• Paper: CodonTransformer: The Global Codon Optimization Benchmark

📁 Dataset Contents

Core Dataset Files

1. finetune_set.json (9.0MB)

Training data for fine-tuning the ColiFormer model

• Format: JSONL with codon-tokenized sequences
• Size: ~4,300 high-CAI E. coli gene sequences
• Fields:
  • idx: Sequence identifier
  • codons: Codon-tokenized DNA sequence (format: AMINO_CODON)
  • organism: Organism ID (51 = Escherichia coli general)
• Usage: Fine-tuning CodonTransformer for E. coli-specific optimization

2. test_set.json (103KB)

Evaluation dataset for model testing

• Format: JSON array of test sequences
• Size: 100 sequences
• Fields:
  • codons: DNA sequence for evaluation
  • organism: Organism ID (51)
• Usage: Performance evaluation and benchmarking

Reference Data for Metrics Calculation

3. ecoli_processed_genes.csv (55MB)

Comprehensive E. coli gene dataset with CAI annotations

• Size: ~50,000 validated E. coli gene sequences
• Fields:
  • gene_id: Gene identifier from NCBI
  • dna_sequence: Complete coding DNA sequence
  • protein_sequence: Translated amino acid sequence
  • cai_score: Calculated Codon Adaptation Index
  • is_high_cai: Boolean flag for high-CAI sequences (used for filtering training data)
• Usage: CAI weight calculation, reference sequences for evaluation metrics

4. CAI.csv (45MB)

Raw CAI scores and sequences

• Fields:
  • gene_id: Gene identifier
  • cai_score: CAI score
  • dna_sequence: DNA sequence
• Usage: Original CAI calculation data

5. Database 3_4300 gene.csv (4.9MB)

High-CAI gene subset

• Size: 4,300 high-quality E. coli genes
• Fields:
  • dna_sequence: High-CAI DNA sequences
• Usage: Identifying high-quality sequences for training

6. organism_tai_weights.json

Organism-specific tRNA Adaptation Index (tAI) weights

• Format: JSON with organism-specific tAI coefficients
• Coverage: Multiple organisms including E. coli
• Usage: Calculating tAI scores for evaluation metrics

📊 Metrics and Evaluation

The dataset enables calculation of multiple codon optimization metrics:

Primary Metrics

• CAI (Codon Adaptation Index): Measures codon usage bias relative to highly expressed genes
• tAI (tRNA Adaptation Index): Reflects tRNA availability for translation
• GC Content: Nucleotide composition analysis

Secondary Metrics

• Restriction Sites: Count of restriction enzyme recognition sites
• Negative Cis Elements: Regulatory sequence analysis
• Homopolymer Runs: Repetitive sequence detection
• ENC (Effective Number of Codons): Codon usage diversity
• CPB (Codon Pair Bias): Codon pair preferences
• SCUO (Synonymous Codon Usage Order): Codon usage ordering

🔬 Model Performance

ColiFormer vs Base Model Results

• CAI Improvement: +6.2% average improvement
• Training Data: 4,300 high-CAI E. coli sequences
• Architecture: BigBird Transformer with Adaptive Learning Methods (ALM)
• Specialization: Optimized specifically for E. coli codon usage patterns

Benchmarking

The dataset includes comprehensive evaluation protocols comparing:

1. Fine-tuned ColiFormer: E. coli-specialized model
2. Base CodonTransformer: General-purpose model
3. Naive HFC: High-frequency codon baseline

🧬 Data Processing Pipeline

1. Data Collection

• Source: NCBI E. coli genome annotations
• Quality filtering: Valid ORFs, proper start/stop codons
• CAI calculation using relative adaptiveness

2. Training Set Creation

• Filter for is_high_cai == True sequences
• Remove duplicates based on DNA sequence
• Format conversion to codon-tokenized representation

3. Test Set Creation

• Sample 100 sequences from lower-CAI pool
• Ensure diversity and representative coverage
• Format for evaluation pipeline

📈 Usage Examples

Loading the Dataset

from datasets import load_dataset

# Load the complete dataset
dataset = load_dataset("saketh11/ColiFormer-Data")

# Load specific files
import pandas as pd
import json

# Training data
with open("finetune_set.json", "r") as f:
    finetune_data = [json.loads(line) for line in f]

# Reference sequences for CAI calculation
processed_genes = pd.read_csv("ecoli_processed_genes.csv")
reference_sequences = processed_genes['dna_sequence'].tolist()

# Calculate CAI weights
from CAI import relative_adaptiveness
cai_weights = relative_adaptiveness(sequences=reference_sequences)

Calculating Metrics

from CAI import CAI
import json

# Load tAI weights
with open("organism_tai_weights.json", "r") as f:
    tai_weights = json.load(f)["Escherichia coli general"]

# Calculate metrics for a sequence
dna_sequence = "ATGAAAGAACTG..."  # Your sequence
cai_score = CAI(dna_sequence, weights=cai_weights)
tai_score = calculate_tAI(dna_sequence, tai_weights)

📚 Citation

If you use this dataset in your research, please cite:

@article{coliformer2024,
  title={ColiFormer: Enhanced E. coli Codon Optimization with Adaptive Learning Methods},
  author={Your Name},
  journal={bioRxiv},
  year={2024},
  note={Fine-tuned model achieving 6.2\% CAI improvement over base CodonTransformer}
}

@article{codontransformer2023,
  title={CodonTransformer: The Global Codon Optimization Benchmark},
  author={Adibvafa Fallahpour and Bartosz Grzybowski and Seyed Pooya Alavizadeh and Ali Emami},
  journal={bioRxiv},
  year={2023},
  doi={10.1101/2023.09.09.556981}
}

🔄 Data Updates

This dataset represents the training and evaluation data used for the initial ColiFormer model. Future updates may include:

• Additional E. coli strains and conditions
• Extended metric calculations
• Comparative analysis with other organisms
• Integration with experimental validation data

⚖️ License

This dataset is released under the MIT License. See LICENSE file for details.

🤝 Contributing

For questions, issues, or contributions related to this dataset, please contact the maintainers or open an issue in the associated model repository.

---

Keywords: codon optimization, E. coli, synthetic biology, protein expression, CAI, tAI, transformer model, bioinformatics