README.md

---
title: ColiFormer Training and Evaluation Dataset
license: mit
tags:
- biology
- codon-optimization
- e-coli
- protein-synthesis
- bioinformatics
- synthetic-biology
size_categories:
- 10K<n<100K
task_categories:
- text-generation
- sequence-modeling
language:
- en
pretty_name: E. coli Codon Optimization Dataset for ColiFormer
---

# ColiFormer Training and Evaluation Dataset

This dataset contains the training and evaluation data used for the **ColiFormer** model - a specialized codon optimization transformer fine-tuned for *Escherichia coli* sequences. The model achieves 6.2% better CAI (Codon Adaptation Index) scores compared to the base CodonTransformer model.

## 🔗 Related Resources

- **Model**: [saketh11/ColiFormer](https://huggingface.co/saketh11/ColiFormer)
- **Base Model**: [adibvafa/CodonTransformer](https://huggingface.co/adibvafa/CodonTransformer)
- **Paper**: [CodonTransformer: The Global Codon Optimization Benchmark](https://www.biorxiv.org/content/10.1101/2023.09.09.556981v1)

## 📁 Dataset Contents

### Core Dataset Files

#### 1. `finetune_set.json` (9.0MB)
**Training data for fine-tuning the ColiFormer model**
- **Format**: JSONL with codon-tokenized sequences
- **Size**: ~4,300 high-CAI E. coli gene sequences
- **Fields**:
  - `idx`: Sequence identifier
  - `codons`: Codon-tokenized DNA sequence (format: `AMINO_CODON`)
  - `organism`: Organism ID (51 = *Escherichia coli* general)
- **Usage**: Fine-tuning CodonTransformer for E. coli-specific optimization

#### 2. `test_set.json` (103KB)
**Evaluation dataset for model testing**
- **Format**: JSON array of test sequences
- **Size**: 100 sequences
- **Fields**:
  - `codons`: DNA sequence for evaluation
  - `organism`: Organism ID (51)
- **Usage**: Performance evaluation and benchmarking

### Reference Data for Metrics Calculation

#### 3. `ecoli_processed_genes.csv` (55MB)
**Comprehensive E. coli gene dataset with CAI annotations**
- **Size**: ~50,000 validated E. coli gene sequences
- **Fields**:
  - `gene_id`: Gene identifier from NCBI
  - `dna_sequence`: Complete coding DNA sequence
  - `protein_sequence`: Translated amino acid sequence
  - `cai_score`: Calculated Codon Adaptation Index
  - `is_high_cai`: Boolean flag for high-CAI sequences (used for filtering training data)
- **Usage**: CAI weight calculation, reference sequences for evaluation metrics

#### 4. `CAI.csv` (45MB)
**Raw CAI scores and sequences**
- **Fields**:
  - `gene_id`: Gene identifier
  - `cai_score`: CAI score
  - `dna_sequence`: DNA sequence
- **Usage**: Original CAI calculation data

#### 5. `Database 3_4300 gene.csv` (4.9MB)
**High-CAI gene subset**
- **Size**: 4,300 high-quality E. coli genes
- **Fields**:
  - `dna_sequence`: High-CAI DNA sequences
- **Usage**: Identifying high-quality sequences for training

#### 6. `organism_tai_weights.json`
**Organism-specific tRNA Adaptation Index (tAI) weights**
- **Format**: JSON with organism-specific tAI coefficients
- **Coverage**: Multiple organisms including *E. coli*
- **Usage**: Calculating tAI scores for evaluation metrics

## 📊 Metrics and Evaluation

The dataset enables calculation of multiple codon optimization metrics:

### Primary Metrics
- **CAI (Codon Adaptation Index)**: Measures codon usage bias relative to highly expressed genes
- **tAI (tRNA Adaptation Index)**: Reflects tRNA availability for translation
- **GC Content**: Nucleotide composition analysis

### Secondary Metrics
- **Restriction Sites**: Count of restriction enzyme recognition sites
- **Negative Cis Elements**: Regulatory sequence analysis
- **Homopolymer Runs**: Repetitive sequence detection
- **ENC (Effective Number of Codons)**: Codon usage diversity
- **CPB (Codon Pair Bias)**: Codon pair preferences
- **SCUO (Synonymous Codon Usage Order)**: Codon usage ordering

## 🔬 Model Performance

### ColiFormer vs Base Model Results
- **CAI Improvement**: +6.2% average improvement
- **Training Data**: 4,300 high-CAI E. coli sequences
- **Architecture**: BigBird Transformer with Adaptive Learning Methods (ALM)
- **Specialization**: Optimized specifically for *E. coli* codon usage patterns

### Benchmarking
The dataset includes comprehensive evaluation protocols comparing:
1. **Fine-tuned ColiFormer**: E. coli-specialized model
2. **Base CodonTransformer**: General-purpose model  
3. **Naive HFC**: High-frequency codon baseline

## 🧬 Data Processing Pipeline

### 1. Data Collection
- Source: NCBI *E. coli* genome annotations
- Quality filtering: Valid ORFs, proper start/stop codons
- CAI calculation using relative adaptiveness

### 2. Training Set Creation
- Filter for `is_high_cai == True` sequences
- Remove duplicates based on DNA sequence
- Format conversion to codon-tokenized representation

### 3. Test Set Creation  
- Sample 100 sequences from lower-CAI pool
- Ensure diversity and representative coverage
- Format for evaluation pipeline

## 📈 Usage Examples

### Loading the Dataset

```python
from datasets import load_dataset

# Load the complete dataset
dataset = load_dataset("saketh11/ColiFormer-Data")

# Load specific files
import pandas as pd
import json

# Training data
with open("finetune_set.json", "r") as f:
    finetune_data = [json.loads(line) for line in f]

# Reference sequences for CAI calculation
processed_genes = pd.read_csv("ecoli_processed_genes.csv")
reference_sequences = processed_genes['dna_sequence'].tolist()

# Calculate CAI weights
from CAI import relative_adaptiveness
cai_weights = relative_adaptiveness(sequences=reference_sequences)
```

### Calculating Metrics

```python
from CAI import CAI
import json

# Load tAI weights
with open("organism_tai_weights.json", "r") as f:
    tai_weights = json.load(f)["Escherichia coli general"]

# Calculate metrics for a sequence
dna_sequence = "ATGAAAGAACTG..."  # Your sequence
cai_score = CAI(dna_sequence, weights=cai_weights)
tai_score = calculate_tAI(dna_sequence, tai_weights)
```

## 📚 Citation

If you use this dataset in your research, please cite:

```bibtex
@article{coliformer2024,
  title={ColiFormer: Enhanced E. coli Codon Optimization with Adaptive Learning Methods},
  author={Your Name},
  journal={bioRxiv},
  year={2024},
  note={Fine-tuned model achieving 6.2\% CAI improvement over base CodonTransformer}
}

@article{codontransformer2023,
  title={CodonTransformer: The Global Codon Optimization Benchmark},
  author={Adibvafa Fallahpour and Bartosz Grzybowski and Seyed Pooya Alavizadeh and Ali Emami},
  journal={bioRxiv},
  year={2023},
  doi={10.1101/2023.09.09.556981}
}
```

## 🔄 Data Updates

This dataset represents the training and evaluation data used for the initial ColiFormer model. Future updates may include:
- Additional E. coli strains and conditions
- Extended metric calculations
- Comparative analysis with other organisms
- Integration with experimental validation data

## ⚖️ License

This dataset is released under the MIT License. See LICENSE file for details.

## 🤝 Contributing

For questions, issues, or contributions related to this dataset, please contact the maintainers or open an issue in the associated model repository.

---

**Keywords**: codon optimization, E. coli, synthetic biology, protein expression, CAI, tAI, transformer model, bioinformatics