# Data Ingestion and Provenance Specification

## Supported contract

The ingestion service accepts UTF-8 `.txt`, `.md`, `.json`, `.jsonl`, and `.csv` files. Each accepted record is normalized into `source`, `content`, `sha256`, and `metadata`. The content digest is calculated after optional redaction so that downstream consumers can identify the exact stored representation.

## Validation rules

The pipeline checks existence, regular-file status, extension allowlists, maximum byte size, UTF-8 decoding, JSON parsing, and CSV header parsing. Unsupported or malformed inputs raise `IngestionError`. When quarantine is enabled, the original bytes are copied to `data/quarantine/` and an audit event records the reason without copying secrets into the event message.

## Provenance requirements

Each production source should provide an origin URL or owner, collection timestamp, license identifier, permitted use, transformation history, and removal/contact procedure. Do not ingest credentials, private personal data, malware, executable payloads, or data whose license does not permit the intended use. The pipeline is passive and never executes files or follows instructions embedded in data.

## Security controls

Credential-like patterns are redacted by default. Redaction is not a substitute for source access control or secret scanning. Operators must scan data before ingestion, restrict filesystem permissions, encrypt sensitive stores, and keep training or evaluation datasets separate from runtime logs.

## Extension procedure

To add a format, implement a parser that returns text records, add a strict extension test, define size and encoding behavior, update this document, and run the complete test suite. Never add a parser that evaluates code, deserializes untrusted objects, or makes network requests during ingestion.

## References

[1]: https://huggingface.co/docs/datasets/en/loading "Hugging Face Datasets Loading"
[2]: https://spdx.dev/learn/handling-license-info/ "SPDX License Information"