code/OpenTSLM/evaluation/baseline/README.md

Time Series LLM Evaluation Framework

This directory contains a modular evaluation framework for testing LLMs on time series datasets. The system is designed to be easily extensible to new datasets and models.

Overview

The evaluation framework consists of:

1. common_evaluator.py - Core evaluation logic that can be reused across datasets
2. evaluate_tsqa.py - TSQA-specific evaluation
3. evaluate_pamap.py - PAMAP-specific evaluation
4. evaluate_all.py - Combined evaluation across all datasets
5. test_baseline.py - Original baseline test (legacy)

Quick Start

Running Individual Dataset Evaluations

# Evaluate on TSQA dataset
python evaluate_tsqa.py

# Evaluate on PAMAP datasets
python evaluate_pamap.py

# Evaluate on all datasets
python evaluate_all.py

Adding New Models

To add new models, simply update the model_names list in any evaluation script:

model_names = [
    "meta-llama/Llama-3.2-1B",
    "google/gemma-3n-e2b",
    "google/gemma-3n-e2b-it",
    "microsoft/DialoGPT-medium",
    "gpt2",
]

Adding New Datasets

To add a new dataset:

1. Create a new evaluation file (e.g., evaluate_newdataset.py)
2. Define an evaluation function that takes (ground_truth, prediction) and returns metrics
3. Import your dataset class and evaluation function in evaluate_all.py

Example evaluation function:

def evaluate_newdataset(ground_truth: str, prediction: str) -> Dict[str, Any]:
    """Evaluate predictions for new dataset."""
    gt_clean = ground_truth.lower().strip()
    pred_clean = prediction.lower().strip()
    
    exact_match = gt_clean == pred_clean
    similarity = SequenceMatcher(None, gt_clean, pred_clean).ratio()
    
    return {
        "exact_match": int(exact_match),
        "similarity": similarity,
        "ground_truth": gt_clean,
        "prediction": pred_clean,
    }

Output Files

The evaluation system generates several output files:

1. Individual Results: evaluation_results_{model}_{dataset}.json - Detailed results for each model-dataset combination
2. Summary CSV: evaluation_results_{timestamp}.csv - Pandas DataFrame with all results
3. Console Output: Real-time progress and summary statistics

Metrics

The framework calculates several metrics for each prediction:

• exact_match: Binary indicator for exact string match
• partial_match: Binary indicator for partial string match
• contains_answer: Binary indicator if prediction contains ground truth
• similarity: Character-level similarity score (0-1)
• has_reasoning: For CoT datasets, indicates if prediction contains reasoning

Configuration

Sample Limits

For faster testing, you can limit the number of samples:

max_samples=50  # Process only first 50 samples
max_samples=None  # Process all samples

Model Parameters

You can customize model parameters:

results_df = evaluator.evaluate_multiple_models(
    model_names=model_names,
    dataset_classes=dataset_classes,
    evaluation_functions=evaluation_functions,
    max_samples=50,
    temperature=0.1,  # Model temperature
    max_new_tokens=100,  # Maximum tokens to generate
)

Dataset-Specific Considerations

TSQADataset

• Extracts answers after "Answer:" in predictions
• Uses built-in time series formatting from dataset class
• Calculates similarity metrics

PAMAP2AccQADataset

• Activity classification from accelerometer data
• Standard exact/partial match metrics

PAMAP2CoTQADataset

• Chain-of-thought reasoning evaluation
• Additional "has_reasoning" metric
• More complex answer extraction

Extending the Framework

Adding Custom Metrics

To add custom metrics, modify the evaluation function:

def evaluate_custom(ground_truth: str, prediction: str) -> Dict[str, Any]:
    # ... existing metrics ...
    
    # Add custom metric
    custom_metric = calculate_custom_metric(ground_truth, prediction)
    
    return {
        # ... existing metrics ...
        "custom_metric": custom_metric,
    }

Adding New Dataset Classes

1. Ensure your dataset class inherits from QADataset
2. Implement the required abstract methods
3. Create an evaluation function
4. Add to the evaluation scripts

Batch Processing

For large-scale evaluations, you can modify the framework to process models in batches or use distributed processing.

Troubleshooting

Common Issues

1. Model Loading Errors: Check if the model name is correct and accessible
2. Memory Issues: Reduce max_samples or use smaller models
3. Import Errors: Ensure all dataset classes are properly imported

Debug Mode

To enable detailed debugging, modify the evaluation scripts to print more information:

# In common_evaluator.py, set debug=True
if idx < 10:  # Print more samples for debugging
    print(f"Sample {idx}: {input_text[:200]}...")

Performance Tips

1. Use GPU: The framework automatically detects and uses CUDA/MPS if available
2. Batch Processing: For multiple models, consider running them in parallel
3. Sample Limiting: Use max_samples for quick testing before full evaluation
4. Model Caching: Models are loaded once per evaluation run

Using OpenAI Models (ChatGPT, GPT-4, etc.)

You can now evaluate OpenAI models (e.g., ChatGPT, GPT-4) using the same evaluation scripts. To do so:

1. Set your OpenAI API key (required):

   export OPENAI_API_KEY=sk-...
   

2. Run the evaluation script with an OpenAI model name prefixed by openai-:

   python evaluate_tsqa.py openai-gpt-4
   python evaluate_pamap.py openai-gpt-3.5-turbo
   

   This will use the OpenAI API instead of a local HuggingFace model.

3. Notes:

   • The model name after openai- should match the OpenAI API model name (e.g., gpt-4, gpt-3.5-turbo).
   • You can adjust max_new_tokens and other parameters as usual.