Spaces:

StrategyStack
/

README

Sleeping

App Files Files Community

README / SCHEMA_VERSIONING.md

stackConsult

Add comprehensive schema versioning and rollback documentation

cc10b69 verified about 1 year ago

preview code

Raw History Blame Contribute Delete

9.58 kB

# Schema Versioning and Rollback Documentation

## Overview

This document establishes the schema versioning strategy and rollback procedures for the activity logging system. It ensures data integrity, backward compatibility, and reliable version control for production deployments.

---

## Current Schema Version: 1.0

### Schema Definition (v1.0)

**File:** `activity_log.csv`  
**Created:** 2025-11-05T10:51:00Z  
**Commit Hash:** 3a67095  

#### Required Fields

| Field | Type | Description | Validation |
|-------|------|-------------|------------|
| `timestamp` | ISO 8601 String | UTC timestamp of activity | Must be valid ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ) |
| `agent_id` | String | Identifier of the agent performing activity | Non-empty string |
| `activity_type` | Enum | Type of activity performed | Must be one of: `init`, `validate`, `ingest`, `update`, `delete`, `query`, `commit`, `rollback` |
| `entity_id` | String | Identifier of entity being acted upon | Non-empty string |
| `status` | Enum | Outcome status of activity | Must be one of: `created`, `success`, `failed`, `in_progress`, `warning`, `error` |
| `notes` | String | Human-readable description | Non-empty string |
| `context` | String | Additional context or metadata | Non-empty string |

#### CSV Format
```csv
timestamp,agent_id,activity_type,entity_id,status,notes,context
2025-11-05T10:51:00Z,comet_assistant,init,activity_log_schema,created,"Canonical CSV schema for activity tracking established","Step 1: Schema and CSV Foundation - initial setup"
```

---

## Version History

### Version 1.0 (Current)
**Date:** 2025-11-05  
**Commit:** 3a67095  
**Status:** Active  

**Changes:**
- Initial schema establishment
- 7 required fields defined
- Atomic ingestion function implemented
- Schema validation with error handling
- Request ID and commit hash tracking

**Files:**
- `activity_log.csv` - Data storage
- `activity_ingestion.py` - Ingestion function with validation
- `SCHEMA_VERSIONING.md` - This documentation

**Migration:** N/A (initial version)

---

## Rollback Procedures

### Git-Based Rollback

All schema changes and data are tracked via Git commits. To roll back to a previous version:

#### 1. Identify Target Commit

```bash
# View commit history
git log --oneline activity_log.csv activity_ingestion.py
# Example output:
# 3a67095 Implement atomic activity ingestion with schema validation
# b1a2c3d Initial activity_log.csv setup
```

#### 2. Rollback Commands

**Option A: Soft Rollback (Review changes before commit)**
```bash
# Reset to previous commit (keeps changes in working directory)
git checkout <commit-hash> -- activity_log.csv activity_ingestion.py
git commit -m "Rollback to schema version X.X (commit <commit-hash>)"
```

**Option B: Hard Rollback (Direct revert)**
```bash
# Revert to specific commit
git revert <commit-hash>
```

**Option C: File-Specific Rollback**
```bash
# Rollback only the CSV data file
git checkout <commit-hash> -- activity_log.csv
git commit -m "Rollback activity_log.csv to <commit-hash>"
```

#### 3. Verify Rollback

```bash
# Check current schema version
grep "SCHEMA_VERSION" activity_ingestion.py
# Validate CSV structure
head -n 5 activity_log.csv
# Run validation tests
python activity_ingestion.py
```

---

## Schema Change Tracking

### Change Log Format

Every schema modification MUST be logged using this structure:

```markdown
### Version X.Y (Status: Active/Deprecated)
**Date:** YYYY-MM-DD  
**Commit:** <git-commit-hash>  
**Status:** Active | Deprecated | Archived  
**Changes:**
- [ADD] New field: `field_name` (type, description)
- [MODIFY] Changed field: `field_name` (old -> new behavior)
- [REMOVE] Deprecated field: `field_name` (reason)
- [BREAKING] Breaking change description
**Migration:**
- Step 1: Migration instructions
- Step 2: Validation procedures
**Rollback Instructions:**
- Specific rollback notes if different from standard procedure
```

### Schema Version Numbering

- **Major Version (X.0):** Breaking changes, incompatible with previous versions
- **Minor Version (X.Y):** New fields added, backward compatible
- **Patch Version (X.Y.Z):** Bug fixes, validation updates, no schema changes

---

## Commit Tracking System

### Request ID Format

Every ingestion operation generates a unique request ID:

```
req_<16-char-hex>
Example: req_a1b2c3d4e5f67890
```

### Commit Hash Generation

Pseudo-commit hashes track each row insertion:

```python
commit_hash = SHA256(request_id + timestamp + agent_id + activity_type)[:16]
```

Example: `3f8d2a1bc5e97604`

### Operation Log

All operations are logged to `activity_ingestion.log`:

```json
{
  "timestamp": "2025-11-05T10:51:00Z",
  "request_id": "req_a1b2c3d4e5f67890",
  "operation": "ingest",
  "status": "success",
  "agent_id": "comet_assistant",
  "activity_type": "validate",
  "error": ""
}
```

---

## Build-Specific Rollback for Different Environments

### Development Build

```bash
# Tag development schema
git tag -a dev-schema-v1.0 -m "Development schema version 1.0"
# Rollback to dev version
git checkout dev-schema-v1.0 -- activity_log.csv activity_ingestion.py
```

### Staging Build

```bash
# Tag staging schema
git tag -a staging-schema-v1.0 -m "Staging schema version 1.0"
# Rollback to staging version
git checkout staging-schema-v1.0 -- activity_log.csv activity_ingestion.py
```

### Production Build

```bash
# Tag production schema (immutable)
git tag -a prod-schema-v1.0 -m "Production schema version 1.0" -s
# Rollback to production version
git checkout prod-schema-v1.0 -- activity_log.csv activity_ingestion.py
# Verify integrity
git verify-tag prod-schema-v1.0
```

---

## Validation Procedures

### Pre-Commit Validation

Before committing any schema changes:

1. **Run Schema Validation Tests**
   ```bash
   python activity_ingestion.py
   ```

2. **Check CSV Integrity**
   ```bash
   # Verify header
   head -n 1 activity_log.csv
   
   # Count rows
   wc -l activity_log.csv
   
   # Check for malformed rows
   awk -F',' 'NF!=7 {print NR, $0}' activity_log.csv
   ```

3. **Validate Timestamp Format**
   ```bash
   # Extract and validate timestamps
   cut -d',' -f1 activity_log.csv | tail -n +2 | while read ts; do
     date -d "$ts" > /dev/null 2>&1 || echo "Invalid: $ts"
   done
   ```

### Post-Rollback Validation

After performing a rollback:

1. Verify schema version matches expected version
2. Run full test suite: `python activity_ingestion.py`
3. Check operation logs for errors
4. Validate data integrity with sample queries
5. Document rollback in change log

---

## Error Recovery

### Corrupted CSV File

```bash
# Restore from last good commit
git checkout HEAD~1 -- activity_log.csv
# Or restore from specific commit
git checkout <commit-hash> -- activity_log.csv
# Verify restoration
python activity_ingestion.py
```

### Failed Schema Migration

```bash
# Immediate rollback
git reset --hard HEAD~1
# Or targeted rollback
git checkout HEAD~1 -- activity_ingestion.py SCHEMA_VERSIONING.md
# Document failure
echo "Migration failed: <reason>" >> SCHEMA_VERSIONING.md
git commit -m "Rollback failed migration - <reason>"
```

### Atomic Operation Failure

The `atomic_append_activity()` function handles failures automatically:

- **Schema Validation Error:** Returns error, no file modification
- **Write Lock Error:** Returns error, temp file cleaned up
- **Partial Write:** Temp file discarded, original file unchanged

All errors logged to `activity_ingestion.log` with request_id for tracing.
---
## Best Practices
### Schema Changes
1. **Always increment version number** when modifying schema
2. **Document all changes** in this file before committing
3. **Create migration scripts** for breaking changes
4. **Test thoroughly** in development before staging/production
5. **Tag releases** with semantic versioning
### Commit Messages
```
[Schema v1.1] Add new field: priority_level

- Added 'priority_level' field (enum: low, medium, high)
- Updated validation function
- Backward compatible with v1.0
- Migration: existing records default to 'medium'
```
### Data Backup
```bash
# Before major changes, create backup
cp activity_log.csv activity_log_backup_$(date +%Y%m%d).csv
# Commit backup to repo
git add activity_log_backup_*.csv
git commit -m "Backup before schema v1.1 migration"
```
---
## Testing Checklist
Before deploying any schema changes:
- [ ] Schema version updated in code
- [ ] Validation rules updated
- [ ] Test suite passes all cases
- [ ] Documentation updated
- [ ] Migration script created (if needed)
- [ ] Rollback procedure documented
- [ ] Staging deployment successful
- [ ] Production backup created
- [ ] Git tag created
---
## Contact & Support
For schema change requests or rollback assistance:
- **Repository:** huggingface.co/spaces/StrategyStack/README
- **Schema Owner:** comet_assistant
- **Last Updated:** 2025-11-05
---
## Appendix: Quick Reference
### Current Version Info
```bash
# Check schema version
grep SCHEMA_VERSION activity_ingestion.py
# View recent commits
git log -5 --oneline
# List all schema tags
git tag -l '*schema*'
```
### Emergency Rollback (One-Liner)
```bash
# Rollback everything to previous commit
git reset --hard HEAD~1 && python activity_ingestion.py
```
### Validate Current State
```bash
# Full validation
python activity_ingestion.py && \
awk -F',' 'NF!=7 {print "ERROR: Line", NR}' activity_log.csv && \
echo "Validation complete"
```
---
**Document Version:** 1.0  
**Last Modified:** 2025-11-05  
**Schema Version:** 1.0  
**Status:** Active

	# Schema Versioning and Rollback Documentation

	## Overview

	This document establishes the schema versioning strategy and rollback procedures for the activity logging system. It ensures data integrity, backward compatibility, and reliable version control for production deployments.

	---

	## Current Schema Version: 1.0

	### Schema Definition (v1.0)

	File: `activity_log.csv`
	Created: 2025-11-05T10:51:00Z
	Commit Hash: 3a67095

	#### Required Fields

	\| Field \| Type \| Description \| Validation \|
	\|-------\|------\|-------------\|------------\|
	\| `timestamp` \| ISO 8601 String \| UTC timestamp of activity \| Must be valid ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ) \|
	\| `agent_id` \| String \| Identifier of the agent performing activity \| Non-empty string \|
	\| `activity_type` \| Enum \| Type of activity performed \| Must be one of: `init`, `validate`, `ingest`, `update`, `delete`, `query`, `commit`, `rollback` \|
	\| `entity_id` \| String \| Identifier of entity being acted upon \| Non-empty string \|
	\| `status` \| Enum \| Outcome status of activity \| Must be one of: `created`, `success`, `failed`, `in_progress`, `warning`, `error` \|
	\| `notes` \| String \| Human-readable description \| Non-empty string \|
	\| `context` \| String \| Additional context or metadata \| Non-empty string \|

	#### CSV Format
	```csv
	timestamp,agent_id,activity_type,entity_id,status,notes,context
	2025-11-05T10:51:00Z,comet_assistant,init,activity_log_schema,created,"Canonical CSV schema for activity tracking established","Step 1: Schema and CSV Foundation - initial setup"
	```

	---

	## Version History

	### Version 1.0 (Current)
	Date: 2025-11-05
	Commit: 3a67095
	Status: Active

	Changes:
	- Initial schema establishment
	- 7 required fields defined
	- Atomic ingestion function implemented
	- Schema validation with error handling
	- Request ID and commit hash tracking

	Files:
	- `activity_log.csv` - Data storage
	- `activity_ingestion.py` - Ingestion function with validation
	- `SCHEMA_VERSIONING.md` - This documentation

	Migration: N/A (initial version)

	---

	## Rollback Procedures

	### Git-Based Rollback

	All schema changes and data are tracked via Git commits. To roll back to a previous version:

	#### 1. Identify Target Commit

	```bash
	# View commit history
	git log --oneline activity_log.csv activity_ingestion.py

	# Example output:
	# 3a67095 Implement atomic activity ingestion with schema validation
	# b1a2c3d Initial activity_log.csv setup
	```

	#### 2. Rollback Commands

	Option A: Soft Rollback (Review changes before commit)
	```bash
	# Reset to previous commit (keeps changes in working directory)
	git checkout <commit-hash> -- activity_log.csv activity_ingestion.py
	git commit -m "Rollback to schema version X.X (commit <commit-hash>)"
	```

	Option B: Hard Rollback (Direct revert)
	```bash
	# Revert to specific commit
	git revert <commit-hash>
	```

	Option C: File-Specific Rollback
	```bash
	# Rollback only the CSV data file
	git checkout <commit-hash> -- activity_log.csv
	git commit -m "Rollback activity_log.csv to <commit-hash>"
	```

	#### 3. Verify Rollback

	```bash
	# Check current schema version
	grep "SCHEMA_VERSION" activity_ingestion.py

	# Validate CSV structure
	head -n 5 activity_log.csv

	# Run validation tests
	python activity_ingestion.py
	```

	---

	## Schema Change Tracking

	### Change Log Format

	Every schema modification MUST be logged using this structure:

	```markdown
	### Version X.Y (Status: Active/Deprecated)
	Date: YYYY-MM-DD
	Commit: <git-commit-hash>
	Status: Active \| Deprecated \| Archived

	Changes:
	- [ADD] New field: `field_name` (type, description)
	- [MODIFY] Changed field: `field_name` (old -> new behavior)
	- [REMOVE] Deprecated field: `field_name` (reason)
	- [BREAKING] Breaking change description

	Migration:
	- Step 1: Migration instructions
	- Step 2: Validation procedures

	Rollback Instructions:
	- Specific rollback notes if different from standard procedure
	```

	### Schema Version Numbering

	- Major Version (X.0): Breaking changes, incompatible with previous versions
	- Minor Version (X.Y): New fields added, backward compatible
	- Patch Version (X.Y.Z): Bug fixes, validation updates, no schema changes

	---

	## Commit Tracking System

	### Request ID Format

	Every ingestion operation generates a unique request ID:

	```
	req_<16-char-hex>

	Example: req_a1b2c3d4e5f67890
	```

	### Commit Hash Generation

	Pseudo-commit hashes track each row insertion:

	```python
	commit_hash = SHA256(request_id + timestamp + agent_id + activity_type)[:16]
	```

	Example: `3f8d2a1bc5e97604`

	### Operation Log

	All operations are logged to `activity_ingestion.log`:

	```json
	{
	"timestamp": "2025-11-05T10:51:00Z",
	"request_id": "req_a1b2c3d4e5f67890",
	"operation": "ingest",
	"status": "success",
	"agent_id": "comet_assistant",
	"activity_type": "validate",
	"error": ""
	}
	```

	---

	## Build-Specific Rollback for Different Environments

	### Development Build

	```bash
	# Tag development schema
	git tag -a dev-schema-v1.0 -m "Development schema version 1.0"

	# Rollback to dev version
	git checkout dev-schema-v1.0 -- activity_log.csv activity_ingestion.py
	```

	### Staging Build

	```bash
	# Tag staging schema
	git tag -a staging-schema-v1.0 -m "Staging schema version 1.0"

	# Rollback to staging version
	git checkout staging-schema-v1.0 -- activity_log.csv activity_ingestion.py
	```

	### Production Build

	```bash
	# Tag production schema (immutable)
	git tag -a prod-schema-v1.0 -m "Production schema version 1.0" -s

	# Rollback to production version
	git checkout prod-schema-v1.0 -- activity_log.csv activity_ingestion.py

	# Verify integrity
	git verify-tag prod-schema-v1.0
	```

	---

	## Validation Procedures

	### Pre-Commit Validation

	Before committing any schema changes:

	1. Run Schema Validation Tests
	```bash
	python activity_ingestion.py
	```

	2. Check CSV Integrity
	```bash
	# Verify header
	head -n 1 activity_log.csv

	# Count rows
	wc -l activity_log.csv

	# Check for malformed rows
	awk -F',' 'NF!=7 {print NR, $0}' activity_log.csv
	```

	3. Validate Timestamp Format
	```bash
	# Extract and validate timestamps
	cut -d',' -f1 activity_log.csv \| tail -n +2 \| while read ts; do
	date -d "$ts" > /dev/null 2>&1 \|\| echo "Invalid: $ts"
	done
	```

	### Post-Rollback Validation

	After performing a rollback:

	1. Verify schema version matches expected version
	2. Run full test suite: `python activity_ingestion.py`
	3. Check operation logs for errors
	4. Validate data integrity with sample queries
	5. Document rollback in change log

	---

	## Error Recovery

	### Corrupted CSV File

	```bash
	# Restore from last good commit
	git checkout HEAD~1 -- activity_log.csv

	# Or restore from specific commit
	git checkout <commit-hash> -- activity_log.csv

	# Verify restoration
	python activity_ingestion.py
	```

	### Failed Schema Migration

	```bash
	# Immediate rollback
	git reset --hard HEAD~1

	# Or targeted rollback
	git checkout HEAD~1 -- activity_ingestion.py SCHEMA_VERSIONING.md

	# Document failure
	echo "Migration failed: <reason>" >> SCHEMA_VERSIONING.md
	git commit -m "Rollback failed migration - <reason>"
	```

	### Atomic Operation Failure

	The `atomic_append_activity()` function handles failures automatically:

	- Schema Validation Error: Returns error, no file modification
	- Write Lock Error: Returns error, temp file cleaned up
	- Partial Write: Temp file discarded, original file unchanged

	All errors logged to `activity_ingestion.log` with request_id for tracing.

	---

	## Best Practices

	### Schema Changes

	1. Always increment version number when modifying schema
	2. Document all changes in this file before committing
	3. Create migration scripts for breaking changes
	4. Test thoroughly in development before staging/production
	5. Tag releases with semantic versioning

	### Commit Messages

	```
	[Schema v1.1] Add new field: priority_level

	- Added 'priority_level' field (enum: low, medium, high)
	- Updated validation function
	- Backward compatible with v1.0
	- Migration: existing records default to 'medium'
	```

	### Data Backup

	```bash
	# Before major changes, create backup
	cp activity_log.csv activity_log_backup_$(date +%Y%m%d).csv

	# Commit backup to repo
	git add activity_log_backup_*.csv
	git commit -m "Backup before schema v1.1 migration"
	```

	---

	## Testing Checklist

	Before deploying any schema changes:

	- [ ] Schema version updated in code
	- [ ] Validation rules updated
	- [ ] Test suite passes all cases
	- [ ] Documentation updated
	- [ ] Migration script created (if needed)
	- [ ] Rollback procedure documented
	- [ ] Staging deployment successful
	- [ ] Production backup created
	- [ ] Git tag created

	---

	## Contact & Support

	For schema change requests or rollback assistance:

	- Repository: huggingface.co/spaces/StrategyStack/README
	- Schema Owner: comet_assistant
	- Last Updated: 2025-11-05

	---

	## Appendix: Quick Reference

	### Current Version Info
	```bash
	# Check schema version
	grep SCHEMA_VERSION activity_ingestion.py

	# View recent commits
	git log -5 --oneline

	# List all schema tags
	git tag -l 'schema'
	```

	### Emergency Rollback (One-Liner)
	```bash
	# Rollback everything to previous commit
	git reset --hard HEAD~1 && python activity_ingestion.py
	```

	### Validate Current State
	```bash
	# Full validation
	python activity_ingestion.py && \
	awk -F',' 'NF!=7 {print "ERROR: Line", NR}' activity_log.csv && \
	echo "Validation complete"
	```

	---

	Document Version: 1.0
	Last Modified: 2025-11-05
	Schema Version: 1.0
	Status: Active