Blame for SCHEMA_VERSIONING.md

# Schema Versioning and Rollback Documentation

## Overview

This document establishes the schema versioning strategy and rollback procedures for the activity logging system. It ensures data integrity, backward compatibility, and reliable version control for production deployments.

---

## Current Schema Version: 1.0

### Schema Definition (v1.0)

**File:** `activity_log.csv`  
**Created:** 2025-11-05T10:51:00Z  
**Commit Hash:** 3a67095  

#### Required Fields

| Field | Type | Description | Validation |
|-------|------|-------------|------------|
| `timestamp` | ISO 8601 String | UTC timestamp of activity | Must be valid ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ) |
| `agent_id` | String | Identifier of the agent performing activity | Non-empty string |
| `activity_type` | Enum | Type of activity performed | Must be one of: `init`, `validate`, `ingest`, `update`, `delete`, `query`, `commit`, `rollback` |
| `entity_id` | String | Identifier of entity being acted upon | Non-empty string |
| `status` | Enum | Outcome status of activity | Must be one of: `created`, `success`, `failed`, `in_progress`, `warning`, `error` |
| `notes` | String | Human-readable description | Non-empty string |
| `context` | String | Additional context or metadata | Non-empty string |

#### CSV Format
```csv
timestamp,agent_id,activity_type,entity_id,status,notes,context
2025-11-05T10:51:00Z,comet_assistant,init,activity_log_schema,created,"Canonical CSV schema for activity tracking established","Step 1: Schema and CSV Foundation - initial setup"
```

---

## Version History

### Version 1.0 (Current)
**Date:** 2025-11-05  
**Commit:** 3a67095  
**Status:** Active  

**Changes:**
- Initial schema establishment
- 7 required fields defined
- Atomic ingestion function implemented
- Schema validation with error handling
- Request ID and commit hash tracking

**Files:**
- `activity_log.csv` - Data storage
- `activity_ingestion.py` - Ingestion function with validation
- `SCHEMA_VERSIONING.md` - This documentation

**Migration:** N/A (initial version)

---

## Rollback Procedures

### Git-Based Rollback

All schema changes and data are tracked via Git commits. To roll back to a previous version:

#### 1. Identify Target Commit

```bash
# View commit history
git log --oneline activity_log.csv activity_ingestion.py

# Example output:
# 3a67095 Implement atomic activity ingestion with schema validation
# b1a2c3d Initial activity_log.csv setup
```

#### 2. Rollback Commands

**Option A: Soft Rollback (Review changes before commit)**
```bash
# Reset to previous commit (keeps changes in working directory)
git checkout <commit-hash> -- activity_log.csv activity_ingestion.py
git commit -m "Rollback to schema version X.X (commit <commit-hash>)"
```

**Option B: Hard Rollback (Direct revert)**
```bash
# Revert to specific commit
git revert <commit-hash>
```

**Option C: File-Specific Rollback**
```bash
# Rollback only the CSV data file
git checkout <commit-hash> -- activity_log.csv
git commit -m "Rollback activity_log.csv to <commit-hash>"
```

#### 3. Verify Rollback

```bash
# Check current schema version
grep "SCHEMA_VERSION" activity_ingestion.py

# Validate CSV structure
head -n 5 activity_log.csv

# Run validation tests
python activity_ingestion.py
```

---

## Schema Change Tracking

### Change Log Format

Every schema modification MUST be logged using this structure:

```markdown
### Version X.Y (Status: Active/Deprecated)
**Date:** YYYY-MM-DD  
**Commit:** <git-commit-hash>  
**Status:** Active | Deprecated | Archived  

**Changes:**
- [ADD] New field: `field_name` (type, description)
- [MODIFY] Changed field: `field_name` (old -> new behavior)
- [REMOVE] Deprecated field: `field_name` (reason)
- [BREAKING] Breaking change description

**Migration:**
- Step 1: Migration instructions
- Step 2: Validation procedures

**Rollback Instructions:**
- Specific rollback notes if different from standard procedure
```

### Schema Version Numbering

- **Major Version (X.0):** Breaking changes, incompatible with previous versions
- **Minor Version (X.Y):** New fields added, backward compatible
- **Patch Version (X.Y.Z):** Bug fixes, validation updates, no schema changes

---

## Commit Tracking System

### Request ID Format

Every ingestion operation generates a unique request ID:

```
req_<16-char-hex>

Example: req_a1b2c3d4e5f67890
```

### Commit Hash Generation

Pseudo-commit hashes track each row insertion:

```python
commit_hash = SHA256(request_id + timestamp + agent_id + activity_type)[:16]
```

Example: `3f8d2a1bc5e97604`

### Operation Log

All operations are logged to `activity_ingestion.log`:

```json
{
  "timestamp": "2025-11-05T10:51:00Z",
  "request_id": "req_a1b2c3d4e5f67890",
  "operation": "ingest",
  "status": "success",
  "agent_id": "comet_assistant",
  "activity_type": "validate",
  "error": ""
}
```

---

## Build-Specific Rollback for Different Environments

### Development Build

```bash
# Tag development schema
git tag -a dev-schema-v1.0 -m "Development schema version 1.0"

# Rollback to dev version
git checkout dev-schema-v1.0 -- activity_log.csv activity_ingestion.py
```

### Staging Build

```bash
# Tag staging schema
git tag -a staging-schema-v1.0 -m "Staging schema version 1.0"

# Rollback to staging version
git checkout staging-schema-v1.0 -- activity_log.csv activity_ingestion.py
```

### Production Build

```bash
# Tag production schema (immutable)
git tag -a prod-schema-v1.0 -m "Production schema version 1.0" -s

# Rollback to production version
git checkout prod-schema-v1.0 -- activity_log.csv activity_ingestion.py

# Verify integrity
git verify-tag prod-schema-v1.0
```

---

## Validation Procedures

### Pre-Commit Validation

Before committing any schema changes:

1. **Run Schema Validation Tests**
   ```bash
   python activity_ingestion.py
   ```

2. **Check CSV Integrity**
   ```bash
   # Verify header
   head -n 1 activity_log.csv
   
   # Count rows
   wc -l activity_log.csv
   
   # Check for malformed rows
   awk -F',' 'NF!=7 {print NR, $0}' activity_log.csv
   ```

3. **Validate Timestamp Format**
   ```bash
   # Extract and validate timestamps
   cut -d',' -f1 activity_log.csv | tail -n +2 | while read ts; do
     date -d "$ts" > /dev/null 2>&1 || echo "Invalid: $ts"
   done
   ```

### Post-Rollback Validation

After performing a rollback:

1. Verify schema version matches expected version
2. Run full test suite: `python activity_ingestion.py`
3. Check operation logs for errors
4. Validate data integrity with sample queries
5. Document rollback in change log

---

## Error Recovery

### Corrupted CSV File

```bash
# Restore from last good commit
git checkout HEAD~1 -- activity_log.csv

# Or restore from specific commit
git checkout <commit-hash> -- activity_log.csv

# Verify restoration
python activity_ingestion.py
```

### Failed Schema Migration

```bash
# Immediate rollback
git reset --hard HEAD~1

# Or targeted rollback
git checkout HEAD~1 -- activity_ingestion.py SCHEMA_VERSIONING.md

# Document failure
echo "Migration failed: <reason>" >> SCHEMA_VERSIONING.md
git commit -m "Rollback failed migration - <reason>"
```

### Atomic Operation Failure

The `atomic_append_activity()` function handles failures automatically:

- **Schema Validation Error:** Returns error, no file modification
- **Write Lock Error:** Returns error, temp file cleaned up
- **Partial Write:** Temp file discarded, original file unchanged

All errors logged to `activity_ingestion.log` with request_id for tracing.

---

## Best Practices

### Schema Changes

1. **Always increment version number** when modifying schema
2. **Document all changes** in this file before committing
3. **Create migration scripts** for breaking changes
4. **Test thoroughly** in development before staging/production
5. **Tag releases** with semantic versioning

### Commit Messages

```
[Schema v1.1] Add new field: priority_level

- Added 'priority_level' field (enum: low, medium, high)
- Updated validation function
- Backward compatible with v1.0
- Migration: existing records default to 'medium'
```

### Data Backup

```bash
# Before major changes, create backup
cp activity_log.csv activity_log_backup_$(date +%Y%m%d).csv

# Commit backup to repo
git add activity_log_backup_*.csv
git commit -m "Backup before schema v1.1 migration"
```

---

## Testing Checklist

Before deploying any schema changes:

- [ ] Schema version updated in code
- [ ] Validation rules updated
- [ ] Test suite passes all cases
- [ ] Documentation updated
- [ ] Migration script created (if needed)
- [ ] Rollback procedure documented
- [ ] Staging deployment successful
- [ ] Production backup created
- [ ] Git tag created

---

## Contact & Support

For schema change requests or rollback assistance:

- **Repository:** huggingface.co/spaces/StrategyStack/README
- **Schema Owner:** comet_assistant
- **Last Updated:** 2025-11-05

---

## Appendix: Quick Reference

### Current Version Info
```bash
# Check schema version
grep SCHEMA_VERSION activity_ingestion.py

# View recent commits
git log -5 --oneline

# List all schema tags
git tag -l '*schema*'
```

### Emergency Rollback (One-Liner)
```bash
# Rollback everything to previous commit
git reset --hard HEAD~1 && python activity_ingestion.py
```

### Validate Current State
```bash
# Full validation
python activity_ingestion.py && \
awk -F',' 'NF!=7 {print "ERROR: Line", NR}' activity_log.csv && \
echo "Validation complete"
```

---

**Document Version:** 1.0  
**Last Modified:** 2025-11-05  
**Schema Version:** 1.0  
**Status:** Active