docs/LOG_RETENTION.md

Log Retention Configuration

Overview

The Attestation Store implements automatic log retention management to prevent unlimited database growth from system logs. Logs older than a configurable retention period are automatically deleted.

Configuration

Environment Variable

Set the log retention period using the LOG_RETENTION_DAYS environment variable:

# Retain logs for 10 days (default)
export LOG_RETENTION_DAYS=10

# Retain logs for 30 days
export LOG_RETENTION_DAYS=30

# Disable automatic cleanup (retain all logs)
export LOG_RETENTION_DAYS=0

Default Behavior

• Default retention period: 10 days
• Cleanup schedule: Runs automatically every 24 hours
• Initial cleanup: Runs once at application startup

How It Works

1. Automatic Cleanup: A background thread runs every 24 hours to delete logs older than the retention period
2. Startup Cleanup: On application startup, an initial cleanup is performed
3. Database Efficiency: Old log entries are permanently deleted from the system_logs table

Manual Cleanup

Using the Cleanup Script

The easiest way to manually clean up logs is using the provided script:

# Use default retention period (from LOG_RETENTION_DAYS)
python scripts/cleanup_logs.py

# Delete logs older than 30 days
python scripts/cleanup_logs.py --days 30

# Preview what would be deleted without actually deleting
python scripts/cleanup_logs.py --dry-run

# Preview with custom retention period
python scripts/cleanup_logs.py --days 60 --dry-run

Using Python Console

You can also trigger log cleanup from within the application context:

from app import app
from logging_utils import cleanup_old_logs

with app.app_context():
    # Use configured retention period (from LOG_RETENTION_DAYS)
    deleted_count = cleanup_old_logs()
    print(f"Deleted {deleted_count} logs")
    
    # Use custom retention period (30 days)
    deleted_count = cleanup_old_logs(retention_days=30)
    print(f"Deleted {deleted_count} logs older than 30 days")
    
    # Disable cleanup for this run
    deleted_count = cleanup_old_logs(retention_days=0)

Note: The cleanup_old_logs function requires a Flask application context because it accesses the database and configuration. Always wrap calls in with app.app_context():.

Log Types Affected

All logs in the system_logs table are subject to retention policies, including:

• Authentication events (login, logout, password changes)
• Attestation operations (upload, download)
• API access logs
• SAML events
• System errors and warnings
• Syslog entries (RFC 5424 format)

Performance Considerations

• Indexed Timestamp: The timestamp column is indexed for efficient cleanup queries
• Batch Deletion: Logs are deleted in a single database transaction
• Non-Blocking: Cleanup runs in a background thread and doesn't block application operations

Monitoring

Cleanup operations are logged at the INFO level:

INFO - Deleted 1523 log entries older than 10 days (before 2025-12-29T10:30:00)
INFO - Scheduled cleanup: removed 42 old log entries

Production Recommendations

Recommended Retention Periods

Environment	Recommended Days	Rationale
Development	7-10 days	Short retention for rapid iteration
Staging	14-30 days	Enough for testing and debugging
Production	30-90 days	Balance compliance needs with storage costs
Compliance-Heavy	90-365 days	Meet regulatory requirements

Best Practices

1. Set Appropriate Retention: Balance compliance requirements with database size
2. Monitor Disk Usage: Ensure your database has adequate storage
3. Export Critical Logs: If long-term retention is needed, export logs to external systems (SIEM, log aggregation)
4. Test Retention Settings: Verify cleanup works as expected in staging before production deployment

Docker/Kubernetes Deployment

Docker Compose

services:
  attstore:
    environment:
      - LOG_RETENTION_DAYS=30

Kubernetes

env:
  - name: LOG_RETENTION_DAYS
    value: "30"

Helm Chart

# values.yaml
config:
  logRetentionDays: 30

Disabling Automatic Cleanup

To disable automatic log cleanup (not recommended for production):

export LOG_RETENTION_DAYS=0

Note: With cleanup disabled, logs will accumulate indefinitely and may impact database performance and storage.

Troubleshooting

Cleanup Not Running

Check application logs for errors:

grep "cleanup" /var/log/attstore/application.log

Database Locks

If cleanup fails due to database locks, it will retry on the next scheduled run (24 hours later).

Verify Configuration

Check the active retention period:

from flask import current_app
config = current_app.config.get('STORAGE_CONFIG')
print(f"Log retention: {config.LOG_RETENTION_DAYS} days")

API Reference

cleanup_old_logs(retention_days=None)

Delete logs older than the specified retention period.

Parameters:

• retention_days (int, optional): Number of days to retain logs. If not provided, uses LOG_RETENTION_DAYS from config. Set to 0 to skip cleanup.

Returns:

• int: Number of log entries deleted

Example:

from logging_utils import cleanup_old_logs
from app import app

with app.app_context():
    # Use default retention period
    deleted = cleanup_old_logs()
    print(f"Deleted {deleted} logs")
    
    # Custom retention period
    deleted = cleanup_old_logs(retention_days=60)
    print(f"Deleted {deleted} logs older than 60 days")

See Also

• System Logging
• Syslog Configuration
• Database Configuration