| # Log Retention Configuration |
|
|
| ## Overview |
|
|
| The Attestation Store implements automatic log retention management to prevent unlimited database growth from system logs. Logs older than a configurable retention period are automatically deleted. |
|
|
| ## Configuration |
|
|
| ### Environment Variable |
|
|
| Set the log retention period using the `LOG_RETENTION_DAYS` environment variable: |
|
|
| ```bash |
| # Retain logs for 10 days (default) |
| export LOG_RETENTION_DAYS=10 |
| |
| # Retain logs for 30 days |
| export LOG_RETENTION_DAYS=30 |
| |
| # Disable automatic cleanup (retain all logs) |
| export LOG_RETENTION_DAYS=0 |
| ``` |
|
|
| ### Default Behavior |
|
|
| - **Default retention period**: 10 days |
| - **Cleanup schedule**: Runs automatically every 24 hours |
| - **Initial cleanup**: Runs once at application startup |
|
|
| ## How It Works |
|
|
| 1. **Automatic Cleanup**: A background thread runs every 24 hours to delete logs older than the retention period |
| 2. **Startup Cleanup**: On application startup, an initial cleanup is performed |
| 3. **Database Efficiency**: Old log entries are permanently deleted from the `system_logs` table |
|
|
| ## Manual Cleanup |
|
|
| ### Using the Cleanup Script |
|
|
| The easiest way to manually clean up logs is using the provided script: |
|
|
| ```bash |
| # Use default retention period (from LOG_RETENTION_DAYS) |
| python scripts/cleanup_logs.py |
| |
| # Delete logs older than 30 days |
| python scripts/cleanup_logs.py --days 30 |
| |
| # Preview what would be deleted without actually deleting |
| python scripts/cleanup_logs.py --dry-run |
| |
| # Preview with custom retention period |
| python scripts/cleanup_logs.py --days 60 --dry-run |
| ``` |
|
|
| ### Using Python Console |
|
|
| You can also trigger log cleanup from within the application context: |
|
|
| ```python |
| from app import app |
| from logging_utils import cleanup_old_logs |
| |
| with app.app_context(): |
| # Use configured retention period (from LOG_RETENTION_DAYS) |
| deleted_count = cleanup_old_logs() |
| print(f"Deleted {deleted_count} logs") |
| |
| # Use custom retention period (30 days) |
| deleted_count = cleanup_old_logs(retention_days=30) |
| print(f"Deleted {deleted_count} logs older than 30 days") |
| |
| # Disable cleanup for this run |
| deleted_count = cleanup_old_logs(retention_days=0) |
| ``` |
|
|
| **Note:** The `cleanup_old_logs` function requires a Flask application context because it accesses the database and configuration. Always wrap calls in `with app.app_context():`. |
|
|
| ## Log Types Affected |
|
|
| All logs in the `system_logs` table are subject to retention policies, including: |
| - Authentication events (login, logout, password changes) |
| - Attestation operations (upload, download) |
| - API access logs |
| - SAML events |
| - System errors and warnings |
| - Syslog entries (RFC 5424 format) |
|
|
| ## Performance Considerations |
|
|
| - **Indexed Timestamp**: The `timestamp` column is indexed for efficient cleanup queries |
| - **Batch Deletion**: Logs are deleted in a single database transaction |
| - **Non-Blocking**: Cleanup runs in a background thread and doesn't block application operations |
|
|
| ## Monitoring |
|
|
| Cleanup operations are logged at the INFO level: |
|
|
| ``` |
| INFO - Deleted 1523 log entries older than 10 days (before 2025-12-29T10:30:00) |
| INFO - Scheduled cleanup: removed 42 old log entries |
| ``` |
|
|
| ## Production Recommendations |
|
|
| ### Recommended Retention Periods |
|
|
| | Environment | Recommended Days | Rationale | |
| |------------|------------------|-----------| |
| | Development | 7-10 days | Short retention for rapid iteration | |
| | Staging | 14-30 days | Enough for testing and debugging | |
| | Production | 30-90 days | Balance compliance needs with storage costs | |
| | Compliance-Heavy | 90-365 days | Meet regulatory requirements | |
|
|
| ### Best Practices |
|
|
| 1. **Set Appropriate Retention**: Balance compliance requirements with database size |
| 2. **Monitor Disk Usage**: Ensure your database has adequate storage |
| 3. **Export Critical Logs**: If long-term retention is needed, export logs to external systems (SIEM, log aggregation) |
| 4. **Test Retention Settings**: Verify cleanup works as expected in staging before production deployment |
|
|
| ## Docker/Kubernetes Deployment |
|
|
| ### Docker Compose |
|
|
| ```yaml |
| services: |
| attstore: |
| environment: |
| - LOG_RETENTION_DAYS=30 |
| ``` |
|
|
| ### Kubernetes |
|
|
| ```yaml |
| env: |
| - name: LOG_RETENTION_DAYS |
| value: "30" |
| ``` |
|
|
| ### Helm Chart |
|
|
| ```yaml |
| # values.yaml |
| config: |
| logRetentionDays: 30 |
| ``` |
|
|
| ## Disabling Automatic Cleanup |
|
|
| To disable automatic log cleanup (not recommended for production): |
|
|
| ```bash |
| export LOG_RETENTION_DAYS=0 |
| ``` |
|
|
| Note: With cleanup disabled, logs will accumulate indefinitely and may impact database performance and storage. |
|
|
| ## Troubleshooting |
|
|
| ### Cleanup Not Running |
|
|
| Check application logs for errors: |
| ```bash |
| grep "cleanup" /var/log/attstore/application.log |
| ``` |
|
|
| ### Database Locks |
|
|
| If cleanup fails due to database locks, it will retry on the next scheduled run (24 hours later). |
|
|
| ### Verify Configuration |
|
|
| Check the active retention period: |
| ```python |
| from flask import current_app |
| config = current_app.config.get('STORAGE_CONFIG') |
| print(f"Log retention: {config.LOG_RETENTION_DAYS} days") |
| ``` |
|
|
| ## API Reference |
|
|
| ### `cleanup_old_logs(retention_days=None)` |
| |
| Delete logs older than the specified retention period. |
| |
| **Parameters:** |
| - `retention_days` (int, optional): Number of days to retain logs. If not provided, uses `LOG_RETENTION_DAYS` from config. Set to 0 to skip cleanup. |
|
|
| **Returns:** |
| - `int`: Number of log entries deleted |
|
|
| **Example:** |
| ```python |
| from logging_utils import cleanup_old_logs |
| from app import app |
| |
| with app.app_context(): |
| # Use default retention period |
| deleted = cleanup_old_logs() |
| print(f"Deleted {deleted} logs") |
| |
| # Custom retention period |
| deleted = cleanup_old_logs(retention_days=60) |
| print(f"Deleted {deleted} logs older than 60 days") |
| ``` |
|
|
| ## See Also |
|
|
| - [System Logging](DOCUMENTATION_INTEGRATION.md#system-logging) |
| - [Syslog Configuration](../README.md#syslog-configuration) |
| - [Database Configuration](../README.md#database-configuration) |
|
|
