| # SPARKNET Security Documentation |
|
|
| This document outlines security considerations, deployment options, and compliance |
| guidelines for the SPARKNET AI-Powered Technology Transfer Office Automation Platform. |
|
|
| ## Overview |
|
|
| SPARKNET handles sensitive data including: |
| - Patent documents and IP information |
| - License agreements and financial terms |
| - Partner/stakeholder contact information |
| - Research data and findings |
|
|
| Proper security measures are essential for production deployments. |
|
|
| --- |
|
|
| ## Deployment Options |
|
|
| ### 1. Fully Local Deployment (Maximum Privacy) |
|
|
| **Recommended for:** Organizations with strict data sovereignty requirements, classified research, or GDPR Article 17 obligations. |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ Your Private Network │ |
| │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ |
| │ │ SPARKNET │──│ Ollama │──│ Local Vector Store │ │ |
| │ │ (Streamlit)│ │ (LLM) │ │ (ChromaDB) │ │ |
| │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ |
| │ │ │ |
| │ ┌─────────────┐ ┌─────────────────────────────────────┐ │ |
| │ │ PostgreSQL │ │ Document Storage (NFS/S3-compat) │ │ |
| │ │ (metadata) │ │ │ │ |
| │ └─────────────┘ └─────────────────────────────────────┘ │ |
| └─────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| **Configuration:** |
| - Set no cloud API keys in `.env` |
| - System automatically uses Ollama for all inference |
| - All data remains within your network |
| - No external API calls for LLM inference |
|
|
| **Setup:** |
| ```bash |
| # Install Ollama |
| curl -fsSL https://ollama.com/install.sh | sh |
| |
| # Pull required models |
| ollama pull llama3.2:latest |
| ollama pull nomic-embed-text |
| |
| # Configure SPARKNET |
| cp .env.example .env |
| # Leave cloud API keys empty |
| |
| # Run |
| streamlit run demo/app.py |
| ``` |
|
|
| ### 2. Hybrid Deployment (Balanced) |
|
|
| **Recommended for:** Organizations that want cloud LLM capabilities for non-sensitive operations while keeping sensitive data local. |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ Your Private Network │ |
| │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ |
| │ │ SPARKNET │──│ Ollama │──│ Document Storage │ │ |
| │ │ (Streamlit)│ │ (Sensitive)│ │ (Encrypted) │ │ |
| │ └──────┬──────┘ └─────────────┘ └─────────────────────┘ │ |
| └─────────│───────────────────────────────────────────────────┘ |
| │ |
| │ (Non-sensitive queries only) |
| ▼ |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ Cloud LLM Providers │ |
| │ ┌─────────┐ ┌─────────┐ ┌─────────────┐ ┌───────────┐ │ |
| │ │ Groq │ │ Gemini │ │ OpenRouter │ │ GitHub │ │ |
| │ └─────────┘ └─────────┘ └─────────────┘ └───────────┘ │ |
| └─────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| **Configuration:** |
| - Configure cloud API keys for general queries |
| - Use document sensitivity classification |
| - Route sensitive documents to local Ollama |
| - Implement data anonymization for cloud queries |
|
|
| ### 3. Cloud Deployment (Streamlit Cloud) |
|
|
| **Recommended for:** Public demos, non-sensitive research, or when local infrastructure is not available. |
|
|
| **Configuration:** |
| ```toml |
| # .streamlit/secrets.toml |
| [auth] |
| password = "your-secure-password" |
| |
| GROQ_API_KEY = "your-key" |
| GOOGLE_API_KEY = "your-key" |
| ``` |
|
|
| **Security Checklist:** |
| - [ ] Use secrets management (never commit API keys) |
| - [ ] Enable authentication |
| - [ ] Review provider data processing policies |
| - [ ] Consider data anonymization |
| - [ ] Implement session timeouts |
|
|
| --- |
|
|
| ## GDPR Compliance |
|
|
| ### Data Processing Principles |
|
|
| SPARKNET is designed to support GDPR compliance: |
|
|
| 1. **Lawfulness, Fairness, Transparency** |
| - Document all data processing activities |
| - Obtain appropriate consent for personal data |
| - Provide clear privacy notices |
|
|
| 2. **Purpose Limitation** |
| - Use data only for stated TTO purposes |
| - Do not repurpose data without consent |
|
|
| 3. **Data Minimization** |
| - Only process necessary data |
| - Anonymize data when possible |
| - Implement data retention policies |
|
|
| 4. **Accuracy** |
| - CriticAgent validation helps ensure accuracy |
| - Human-in-the-loop for critical decisions |
| - Source verification for claims |
|
|
| 5. **Storage Limitation** |
| - Configure `DATA_RETENTION_DAYS` in `.env` |
| - Implement automatic data purging |
| - Support data deletion requests |
|
|
| 6. **Integrity and Confidentiality** |
| - Encrypt data at rest |
| - Use TLS for data in transit |
| - Implement access controls |
|
|
| ### Data Subject Rights |
|
|
| Support for GDPR data subject rights: |
|
|
| | Right | Implementation | |
| |-------|----------------| |
| | Access | Export function for user data | |
| | Rectification | Edit capabilities in UI | |
| | Erasure | Delete user data on request | |
| | Portability | JSON/CSV export options | |
| | Objection | Opt-out from AI processing | |
|
|
| ### Cross-Border Data Transfers |
|
|
| When using cloud LLM providers: |
|
|
| 1. **EU-US Data Transfers:** |
| - Review provider's Data Processing Agreement |
| - Ensure Standard Contractual Clauses in place |
| - Consider EU-hosted alternatives |
|
|
| 2. **Recommended Approach:** |
| - Use Ollama for EU data residency |
| - Anonymize data before cloud API calls |
| - Implement geographic routing |
|
|
| --- |
|
|
| ## Security Best Practices |
|
|
| ### API Key Management |
|
|
| ```python |
| # GOOD: Load from environment/secrets |
| api_key = os.environ.get("GROQ_API_KEY") |
| # or |
| api_key = st.secrets.get("GROQ_API_KEY") |
| |
| # BAD: Hardcoded keys |
| api_key = "gsk_abc123..." # NEVER DO THIS |
| ``` |
|
|
| ### Authentication |
|
|
| Configure authentication in `.streamlit/secrets.toml`: |
|
|
| ```toml |
| [auth] |
| # Single user |
| password = "strong-password-here" |
| |
| # Multi-user |
| [auth.users] |
| admin = "admin-password" |
| analyst = "analyst-password" |
| viewer = "viewer-password" |
| ``` |
|
|
| ### Audit Logging |
|
|
| Enable audit logging for compliance: |
|
|
| ```env |
| AUDIT_LOG_ENABLED=true |
| AUDIT_LOG_PATH=./logs/audit.log |
| ``` |
|
|
| Audit log includes: |
| - User authentication events |
| - Document access |
| - AI query/response pairs |
| - Decision point approvals |
|
|
| ### Network Security |
|
|
| For production deployments: |
|
|
| 1. **Firewall Rules:** |
| - Restrict Ollama to internal network |
| - Limit database access to app servers |
| - Use VPN for remote access |
|
|
| 2. **TLS/SSL:** |
| - Enable HTTPS for Streamlit |
| - Use encrypted database connections |
| - Secure WebSocket connections |
|
|
| 3. **Access Control:** |
| - Implement role-based access |
| - Use IP allowlisting where possible |
| - Enable MFA for admin access |
|
|
| --- |
|
|
| ## Sensitive Data Handling |
|
|
| ### Document Classification |
|
|
| SPARKNET can classify documents by sensitivity: |
|
|
| | Level | Description | Handling | |
| |-------|-------------|----------| |
| | Public | Non-confidential | Cloud LLM allowed | |
| | Internal | Business confidential | Prefer local | |
| | Confidential | Sensitive business | Local only | |
| | Restricted | Highly sensitive | Local + encryption | |
|
|
| ### PII Detection |
|
|
| Enable PII detection: |
|
|
| ```env |
| PII_DETECTION_ENABLED=true |
| ``` |
|
|
| Detected PII types: |
| - Names (persons) |
| - Email addresses |
| - Phone numbers |
| - Addresses |
| - ID numbers |
|
|
| ### Data Anonymization |
|
|
| For cloud API calls, implement anonymization: |
|
|
| ```python |
| # Pseudonymization example |
| text = text.replace(real_name, "[PERSON_1]") |
| text = text.replace(company_name, "[COMPANY_1]") |
| ``` |
|
|
| --- |
|
|
| ## Incident Response |
|
|
| ### Security Incident Procedure |
|
|
| 1. **Detection:** Monitor audit logs and alerts |
| 2. **Containment:** Isolate affected systems |
| 3. **Investigation:** Determine scope and impact |
| 4. **Notification:** Inform stakeholders (72h for GDPR) |
| 5. **Recovery:** Restore from clean backups |
| 6. **Lessons Learned:** Update security measures |
|
|
| ### Contact |
|
|
| For security issues: |
| - Review issue privately before public disclosure |
| - Report to project maintainers |
| - Follow responsible disclosure practices |
|
|
| --- |
|
|
| ## Compliance Checklist |
|
|
| ### Pre-Deployment |
|
|
| - [ ] API keys stored in secrets management |
| - [ ] Authentication configured |
| - [ ] Audit logging enabled |
| - [ ] Data retention policy defined |
| - [ ] Backup strategy implemented |
| - [ ] Network security reviewed |
|
|
| ### GDPR Compliance |
|
|
| - [ ] Data processing register updated |
| - [ ] Privacy notice published |
| - [ ] Data subject rights procedures in place |
| - [ ] Cross-border transfer safeguards |
| - [ ] Data Protection Impact Assessment (if required) |
|
|
| ### Ongoing |
|
|
| - [ ] Regular security audits |
| - [ ] Log review and monitoring |
| - [ ] Access control review |
| - [ ] Incident response testing |
| - [ ] Staff security training |
|
|
| --- |
|
|
| ## Additional Resources |
|
|
| - [GDPR Official Text](https://gdpr.eu/) |
| - [Ollama Documentation](https://ollama.com/) |
| - [Streamlit Security](https://docs.streamlit.io/deploy/streamlit-community-cloud/security) |
| - [OWASP Top 10](https://owasp.org/Top10/) |
|
|
| --- |
|
|
| *SPARKNET - VISTA/Horizon EU Project* |
| *Last Updated: 2025* |
|
|
