Blame for docs/LOG_RETENTION.md

# Log Retention Configuration

## Overview

The Attestation Store implements automatic log retention management to prevent unlimited database growth from system logs. Logs older than a configurable retention period are automatically deleted.

## Configuration

### Environment Variable

Set the log retention period using the `LOG_RETENTION_DAYS` environment variable:

```bash
# Retain logs for 10 days (default)
export LOG_RETENTION_DAYS=10

# Retain logs for 30 days
export LOG_RETENTION_DAYS=30

# Disable automatic cleanup (retain all logs)
export LOG_RETENTION_DAYS=0
```

### Default Behavior

- **Default retention period**: 10 days
- **Cleanup schedule**: Runs automatically every 24 hours
- **Initial cleanup**: Runs once at application startup

## How It Works

1. **Automatic Cleanup**: A background thread runs every 24 hours to delete logs older than the retention period
2. **Startup Cleanup**: On application startup, an initial cleanup is performed
3. **Database Efficiency**: Old log entries are permanently deleted from the `system_logs` table

## Manual Cleanup

### Using the Cleanup Script

The easiest way to manually clean up logs is using the provided script:

```bash
# Use default retention period (from LOG_RETENTION_DAYS)
python scripts/cleanup_logs.py

# Delete logs older than 30 days
python scripts/cleanup_logs.py --days 30

# Preview what would be deleted without actually deleting
python scripts/cleanup_logs.py --dry-run

# Preview with custom retention period
python scripts/cleanup_logs.py --days 60 --dry-run
```

### Using Python Console

You can also trigger log cleanup from within the application context:

```python
from app import app
from logging_utils import cleanup_old_logs

with app.app_context():
    # Use configured retention period (from LOG_RETENTION_DAYS)
    deleted_count = cleanup_old_logs()
    print(f"Deleted {deleted_count} logs")
    
    # Use custom retention period (30 days)
    deleted_count = cleanup_old_logs(retention_days=30)
    print(f"Deleted {deleted_count} logs older than 30 days")
    
    # Disable cleanup for this run
    deleted_count = cleanup_old_logs(retention_days=0)
```

**Note:** The `cleanup_old_logs` function requires a Flask application context because it accesses the database and configuration. Always wrap calls in `with app.app_context():`.

## Log Types Affected

All logs in the `system_logs` table are subject to retention policies, including:
- Authentication events (login, logout, password changes)
- Attestation operations (upload, download)
- API access logs
- SAML events
- System errors and warnings
- Syslog entries (RFC 5424 format)

## Performance Considerations

- **Indexed Timestamp**: The `timestamp` column is indexed for efficient cleanup queries
- **Batch Deletion**: Logs are deleted in a single database transaction
- **Non-Blocking**: Cleanup runs in a background thread and doesn't block application operations

## Monitoring

Cleanup operations are logged at the INFO level:

```
INFO - Deleted 1523 log entries older than 10 days (before 2025-12-29T10:30:00)
INFO - Scheduled cleanup: removed 42 old log entries
```

## Production Recommendations

### Recommended Retention Periods

| Environment | Recommended Days | Rationale |
|------------|------------------|-----------|
| Development | 7-10 days | Short retention for rapid iteration |
| Staging | 14-30 days | Enough for testing and debugging |
| Production | 30-90 days | Balance compliance needs with storage costs |
| Compliance-Heavy | 90-365 days | Meet regulatory requirements |

### Best Practices

1. **Set Appropriate Retention**: Balance compliance requirements with database size
2. **Monitor Disk Usage**: Ensure your database has adequate storage
3. **Export Critical Logs**: If long-term retention is needed, export logs to external systems (SIEM, log aggregation)
4. **Test Retention Settings**: Verify cleanup works as expected in staging before production deployment

## Docker/Kubernetes Deployment

### Docker Compose

```yaml
services:
  attstore:
    environment:
      - LOG_RETENTION_DAYS=30
```

### Kubernetes

```yaml
env:
  - name: LOG_RETENTION_DAYS
    value: "30"
```

### Helm Chart

```yaml
# values.yaml
config:
  logRetentionDays: 30
```

## Disabling Automatic Cleanup

To disable automatic log cleanup (not recommended for production):

```bash
export LOG_RETENTION_DAYS=0
```

Note: With cleanup disabled, logs will accumulate indefinitely and may impact database performance and storage.

## Troubleshooting

### Cleanup Not Running

Check application logs for errors:
```bash
grep "cleanup" /var/log/attstore/application.log
```

### Database Locks

If cleanup fails due to database locks, it will retry on the next scheduled run (24 hours later).

### Verify Configuration

Check the active retention period:
```python
from flask import current_app
config = current_app.config.get('STORAGE_CONFIG')
print(f"Log retention: {config.LOG_RETENTION_DAYS} days")
```

## API Reference

### `cleanup_old_logs(retention_days=None)`

Delete logs older than the specified retention period.

**Parameters:**
- `retention_days` (int, optional): Number of days to retain logs. If not provided, uses `LOG_RETENTION_DAYS` from config. Set to 0 to skip cleanup.

**Returns:**
- `int`: Number of log entries deleted

**Example:**
```python
from logging_utils import cleanup_old_logs
from app import app

with app.app_context():
    # Use default retention period
    deleted = cleanup_old_logs()
    print(f"Deleted {deleted} logs")
    
    # Custom retention period
    deleted = cleanup_old_logs(retention_days=60)
    print(f"Deleted {deleted} logs older than 60 days")
```

## See Also

- [System Logging](DOCUMENTATION_INTEGRATION.md#system-logging)
- [Syslog Configuration](../README.md#syslog-configuration)
- [Database Configuration](../README.md#database-configuration)