SCHEMA_VERSIONING.md

Schema Versioning and Rollback Documentation

Overview

This document establishes the schema versioning strategy and rollback procedures for the activity logging system. It ensures data integrity, backward compatibility, and reliable version control for production deployments.

---

Current Schema Version: 1.0

Schema Definition (v1.0)

File: activity_log.csv
Created: 2025-11-05T10:51:00Z
Commit Hash: 3a67095

Required Fields

Field	Type	Description	Validation
timestamp	ISO 8601 String	UTC timestamp of activity	Must be valid ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ)
agent_id	String	Identifier of the agent performing activity	Non-empty string
activity_type	Enum	Type of activity performed	Must be one of: init, validate, ingest, update, delete, query, commit, rollback
entity_id	String	Identifier of entity being acted upon	Non-empty string
status	Enum	Outcome status of activity	Must be one of: created, success, failed, in_progress, warning, error
notes	String	Human-readable description	Non-empty string
context	String	Additional context or metadata	Non-empty string

CSV Format

timestamp,agent_id,activity_type,entity_id,status,notes,context
2025-11-05T10:51:00Z,comet_assistant,init,activity_log_schema,created,"Canonical CSV schema for activity tracking established","Step 1: Schema and CSV Foundation - initial setup"

---

Version History

Version 1.0 (Current)

Date: 2025-11-05
Commit: 3a67095
Status: Active

Changes:

• Initial schema establishment
• 7 required fields defined
• Atomic ingestion function implemented
• Schema validation with error handling
• Request ID and commit hash tracking

Files:

• activity_log.csv - Data storage
• activity_ingestion.py - Ingestion function with validation
• SCHEMA_VERSIONING.md - This documentation

Migration: N/A (initial version)

---

Rollback Procedures

Git-Based Rollback

All schema changes and data are tracked via Git commits. To roll back to a previous version:

1. Identify Target Commit

# View commit history
git log --oneline activity_log.csv activity_ingestion.py

# Example output:
# 3a67095 Implement atomic activity ingestion with schema validation
# b1a2c3d Initial activity_log.csv setup

2. Rollback Commands

Option A: Soft Rollback (Review changes before commit)

# Reset to previous commit (keeps changes in working directory)
git checkout <commit-hash> -- activity_log.csv activity_ingestion.py
git commit -m "Rollback to schema version X.X (commit <commit-hash>)"

Option B: Hard Rollback (Direct revert)

# Revert to specific commit
git revert <commit-hash>

Option C: File-Specific Rollback

# Rollback only the CSV data file
git checkout <commit-hash> -- activity_log.csv
git commit -m "Rollback activity_log.csv to <commit-hash>"

3. Verify Rollback

# Check current schema version
grep "SCHEMA_VERSION" activity_ingestion.py

# Validate CSV structure
head -n 5 activity_log.csv

# Run validation tests
python activity_ingestion.py

---

Schema Change Tracking

Change Log Format

Every schema modification MUST be logged using this structure:

### Version X.Y (Status: Active/Deprecated)
**Date:** YYYY-MM-DD  
**Commit:** <git-commit-hash>  
**Status:** Active | Deprecated | Archived  

**Changes:**
- [ADD] New field: `field_name` (type, description)
- [MODIFY] Changed field: `field_name` (old -> new behavior)
- [REMOVE] Deprecated field: `field_name` (reason)
- [BREAKING] Breaking change description

**Migration:**
- Step 1: Migration instructions
- Step 2: Validation procedures

**Rollback Instructions:**
- Specific rollback notes if different from standard procedure

Schema Version Numbering

• Major Version (X.0): Breaking changes, incompatible with previous versions
• Minor Version (X.Y): New fields added, backward compatible
• Patch Version (X.Y.Z): Bug fixes, validation updates, no schema changes

---

Commit Tracking System

Request ID Format

Every ingestion operation generates a unique request ID:

req_<16-char-hex>

Example: req_a1b2c3d4e5f67890

Commit Hash Generation

Pseudo-commit hashes track each row insertion:

commit_hash = SHA256(request_id + timestamp + agent_id + activity_type)[:16]

Example: 3f8d2a1bc5e97604

Operation Log

All operations are logged to activity_ingestion.log:

{
  "timestamp": "2025-11-05T10:51:00Z",
  "request_id": "req_a1b2c3d4e5f67890",
  "operation": "ingest",
  "status": "success",
  "agent_id": "comet_assistant",
  "activity_type": "validate",
  "error": ""
}

---

Build-Specific Rollback for Different Environments

Development Build

# Tag development schema
git tag -a dev-schema-v1.0 -m "Development schema version 1.0"

# Rollback to dev version
git checkout dev-schema-v1.0 -- activity_log.csv activity_ingestion.py

Staging Build

# Tag staging schema
git tag -a staging-schema-v1.0 -m "Staging schema version 1.0"

# Rollback to staging version
git checkout staging-schema-v1.0 -- activity_log.csv activity_ingestion.py

Production Build

# Tag production schema (immutable)
git tag -a prod-schema-v1.0 -m "Production schema version 1.0" -s

# Rollback to production version
git checkout prod-schema-v1.0 -- activity_log.csv activity_ingestion.py

# Verify integrity
git verify-tag prod-schema-v1.0

---

Validation Procedures

Pre-Commit Validation

Before committing any schema changes:

1. Run Schema Validation Tests

   python activity_ingestion.py
   

2. Check CSV Integrity

   # Verify header
   head -n 1 activity_log.csv
   
   # Count rows
   wc -l activity_log.csv
   
   # Check for malformed rows
   awk -F',' 'NF!=7 {print NR, $0}' activity_log.csv
   

3. Validate Timestamp Format

   # Extract and validate timestamps
   cut -d',' -f1 activity_log.csv | tail -n +2 | while read ts; do
     date -d "$ts" > /dev/null 2>&1 || echo "Invalid: $ts"
   done
   

Post-Rollback Validation

After performing a rollback:

1. Verify schema version matches expected version
2. Run full test suite: python activity_ingestion.py
3. Check operation logs for errors
4. Validate data integrity with sample queries
5. Document rollback in change log

---

Error Recovery

Corrupted CSV File

# Restore from last good commit
git checkout HEAD~1 -- activity_log.csv

# Or restore from specific commit
git checkout <commit-hash> -- activity_log.csv

# Verify restoration
python activity_ingestion.py

Failed Schema Migration

# Immediate rollback
git reset --hard HEAD~1

# Or targeted rollback
git checkout HEAD~1 -- activity_ingestion.py SCHEMA_VERSIONING.md

# Document failure
echo "Migration failed: <reason>" >> SCHEMA_VERSIONING.md
git commit -m "Rollback failed migration - <reason>"

Atomic Operation Failure

The atomic_append_activity() function handles failures automatically:

• Schema Validation Error: Returns error, no file modification
• Write Lock Error: Returns error, temp file cleaned up
• Partial Write: Temp file discarded, original file unchanged

All errors logged to activity_ingestion.log with request_id for tracing.

---

Best Practices

Schema Changes

1. Always increment version number when modifying schema
2. Document all changes in this file before committing
3. Create migration scripts for breaking changes
4. Test thoroughly in development before staging/production
5. Tag releases with semantic versioning

Commit Messages

[Schema v1.1] Add new field: priority_level

- Added 'priority_level' field (enum: low, medium, high)
- Updated validation function
- Backward compatible with v1.0
- Migration: existing records default to 'medium'

Data Backup

# Before major changes, create backup
cp activity_log.csv activity_log_backup_$(date +%Y%m%d).csv

# Commit backup to repo
git add activity_log_backup_*.csv
git commit -m "Backup before schema v1.1 migration"

---

Testing Checklist

Before deploying any schema changes:

• Schema version updated in code
• Validation rules updated
• Test suite passes all cases
• Documentation updated
• Migration script created (if needed)
• Rollback procedure documented
• Staging deployment successful
• Production backup created
• Git tag created

---

Contact & Support

For schema change requests or rollback assistance:

• Repository: huggingface.co/spaces/StrategyStack/README
• Schema Owner: comet_assistant
• Last Updated: 2025-11-05

---

Appendix: Quick Reference

Current Version Info

# Check schema version
grep SCHEMA_VERSION activity_ingestion.py

# View recent commits
git log -5 --oneline

# List all schema tags
git tag -l '*schema*'

Emergency Rollback (One-Liner)

# Rollback everything to previous commit
git reset --hard HEAD~1 && python activity_ingestion.py

Validate Current State

# Full validation
python activity_ingestion.py && \
awk -F',' 'NF!=7 {print "ERROR: Line", NR}' activity_log.csv && \
echo "Validation complete"

---

Document Version: 1.0
Last Modified: 2025-11-05
Schema Version: 1.0
Status: Active