document-qa-system / create_sample_docs.py
Varun2007's picture
Production Document Question Answering (DQA) ready for Hugging Face Spaces
d3d09d6
Raw History Blame Contribute Delete
15.8 kB
"""Script to generate realistic test documents:
1. data/sample_financial_report.pdf (Multi-page PDF with headers and tables)
2. data/sample_cloud_architecture.docx (DOCX with styles, headings, tables)
3. data/sample_security_policy.txt (Markdown/TXT with clear sections)
"""
import os
from reportlab.lib.pagesizes import letter
from reportlab.lib import colors
from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle
from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, Table, TableStyle, PageBreak
import docx
from docx.shared import Inches, Pt, RGBColor
def generate_pdf(output_path: str):
doc = SimpleDocTemplate(
output_path,
pagesize=letter,
rightMargin=54, leftMargin=54,
topMargin=54, bottomMargin=54
)
styles = getSampleStyleSheet()
title_style = ParagraphStyle(
'DocTitle',
parent=styles['Heading1'],
fontSize=20,
leading=24,
textColor=colors.HexColor('#1a365d'),
spaceAfter=12
)
h1_style = ParagraphStyle(
'SectionH1',
parent=styles['Heading2'],
fontSize=14,
leading=18,
textColor=colors.HexColor('#2b6cb0'),
spaceBefore=12,
spaceAfter=6
)
body_style = ParagraphStyle(
'DocBody',
parent=styles['BodyText'],
fontSize=10,
leading=14,
textColor=colors.HexColor('#2d3748'),
spaceAfter=8
)
story = []
# Page 1: Executive Summary & Financial Highlights
story.append(Paragraph("Acme Global Enterprises - Q3 Financial Report", title_style))
story.append(Paragraph("1. Executive Summary", h1_style))
story.append(Paragraph(
"In the third quarter of fiscal year 2023, Acme Global Enterprises delivered robust financial "
"results across all operating segments. Total consolidated revenue reached $4.2 billion, representing "
"a 14.8% increase year-over-year. Operating income expanded to $850 million, resulting in an operating "
"margin of 20.2%. Net income for the quarter stood at $620 million, yielding a net profit margin of 18.4%. "
"Diluted earnings per share (EPS) grew to $2.15 compared to $1.82 in the prior year period.",
body_style
))
story.append(Paragraph(
"Cash flow from operating activities was $920 million, while free cash flow reached $710 million after "
"capital expenditures of $210 million. The Board of Directors declared a quarterly cash dividend of $0.45 "
"per common share and authorized an expansion of the share repurchase program by $500 million.",
body_style
))
story.append(Spacer(1, 14))
story.append(Paragraph("2. Regional Performance Breakdown", h1_style))
story.append(Paragraph(
"North America remained our primary revenue contributor, generating $2.18 billion (52% of total revenue) "
"driven by high demand in enterprise cloud services and cybersecurity engagements. "
"Europe, Middle East, and Africa (EMEA) generated $1.18 billion (28% of total revenue), displaying resilience "
"despite currency headwinds from the Euro and British Pound. "
"Asia-Pacific (APAC) contributed $840 million (20% of total revenue), recording the highest regional growth "
"rate at 22.4% year-over-year, spearheaded by market expansion in Japan and India.",
body_style
))
# Page 2: Segment Breakdown Table & R&D
story.append(PageBreak())
story.append(Paragraph("3. Operating Segment Financials", h1_style))
story.append(Paragraph(
"The following table summarizes the financial performance across Acme's three primary operating business units for Q3:",
body_style
))
table_data = [
["Operating Segment", "Q3 Revenue ($M)", "YoY Growth (%)", "Gross Margin (%)", "Operating Margin (%)"],
["Cloud & AI Solutions", "2,150", "+24.5%", "68.2%", "28.4%"],
["Enterprise Hardware", "1,250", "+3.2%", "41.5%", "14.1%"],
["Consulting & Managed Services", "800", "+8.9%", "48.0%", "17.5%"],
["Total Consolidated", "4,200", "+14.8%", "56.4%", "20.2%"],
]
t = Table(table_data, colWidths=[160, 85, 80, 85, 90])
t.setStyle(TableStyle([
('BACKGROUND', (0, 0), (-1, 0), colors.HexColor('#2b6cb0')),
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
('ALIGN', (0, 0), (0, -1), 'LEFT'),
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
('FONTSIZE', (0, 0), (-1, -1), 9),
('BOTTOMPADDING', (0, 0), (-1, 0), 6),
('TOPPADDING', (0, 0), (-1, 0), 6),
('BACKGROUND', (0, -1), (-1, -1), colors.HexColor('#e2e8f0')),
('FONTNAME', (0, -1), (-1, -1), 'Helvetica-Bold'),
('GRID', (0, 0), (-1, -1), 0.5, colors.HexColor('#cbd5e0')),
('ROWBACKGROUNDS', (0, 1), (-1, -2), [colors.white, colors.HexColor('#f7fafc')])
]))
story.append(t)
story.append(Spacer(1, 14))
story.append(Paragraph("4. Research & Development Investment", h1_style))
story.append(Paragraph(
"Acme invested $410 million in Research and Development during Q3, representing 9.8% of net revenue. "
"Key focus areas included next-generation transformer inference optimization, confidential cloud computing, "
"and energy-efficient hardware acceleration architectures. During the quarter, the company was granted 47 new "
"patents by the USPTO, bringing its global patent portfolio to over 1,280 active patents.",
body_style
))
# Page 3: Risk Factors & Full Year Guidance
story.append(PageBreak())
story.append(Paragraph("5. Risk Factors and Regulatory Environment", h1_style))
story.append(Paragraph(
"Management has identified several operational and external risk factors affecting upcoming fiscal quarters: "
"1. Supply chain constraints in semiconductor packaging and advanced memory modules could extend lead times for Enterprise Hardware by 4 to 6 weeks. "
"2. Foreign currency volatility, particularly relative fluctuations in EUR/USD and JPY/USD exchange rates, could impact reported operating margins by 50 to 80 basis points. "
"3. Compliance preparations for the European Union Artificial Intelligence Act (EU AI Act) are currently underway; compliance investments are projected at $25 million across FY2024.",
body_style
))
story.append(Spacer(1, 12))
story.append(Paragraph("6. Full Year Guidance and Capital Allocation", h1_style))
story.append(Paragraph(
"For the full fiscal year 2024, Acme Global Enterprises is raising its consolidated revenue guidance to $17.5 billion "
"(range of $17.4B to $17.6B), reflecting expected annual growth of 13.5%. Full-year operating margin is anticipated to "
"remain between 19.5% and 20.5%. Capital expenditures for FY2024 are budgeted at $850 million, focused on data center expansions "
"in Ireland and Singapore. Total share repurchases for the fiscal year are targeted at $1.2 billion, subject to macroeconomic conditions.",
body_style
))
doc.build(story)
print(f"Generated PDF: {output_path}")
def generate_docx(output_path: str):
doc = docx.Document()
# Title
doc.add_heading("Cloud Platform Architecture & Infrastructure Specification", level=0)
# 1. Overview
doc.add_heading("1. Architecture Overview", level=1)
doc.add_paragraph(
"The NexaScale Cloud Platform is designed as an active-active, multi-region distributed system deployed "
"across AWS (us-east-1 and us-west-2) and Google Cloud Platform (us-central1). The architecture adheres to "
"the 12-factor application methodology and leverages Kubernetes (EKS and GKE) for stateless workload container "
"orchestration. Ingress traffic is managed via Cloudflare Enterprise with Anycast routing and automated DDoS mitigation."
)
# 2. Network Topography
doc.add_heading("2. Network Topography and Interconnects", level=1)
doc.add_paragraph(
"VPCs in us-east-1 use the CIDR block 10.100.0.0/16, while us-west-2 utilizes 10.101.0.0/16. "
"GCP VPC networks use the 10.200.0.0/16 address space. Hybrid cloud and cross-cloud communications are facilitated "
"through dual redundant AWS Direct Connect 10 Gbps dedicated circuits and Google Cloud Interconnect partner attachments. "
"Cross-VPC routing within AWS utilizes AWS Transit Gateway with full route segmentation separating production, staging, "
"and management VPCs."
)
# 3. Database Tier
doc.add_heading("3. Database Tier and Caching Layer", level=1)
doc.add_paragraph(
"The transactional persistence layer is powered by Amazon Aurora PostgreSQL (Serverless v2) configured in Multi-Region "
"Global Database mode. Write operations are directed to the primary cluster in us-east-1, while read operations are served locally "
"by read replicas in us-west-2 and us-central1 with an average replication lag of less than 12 milliseconds. "
"For high-speed low-latency caching, Redis Cluster (v7.2) is deployed with cluster mode enabled across 6 shards (3 primary, 3 replica). "
"The cache invalidation strategy employs event-driven Kafka message streams with a time-to-live (TTL) of 3600 seconds on user sessions."
)
# 4. Service Level Objectives Table
doc.add_heading("4. Service Level Objectives (SLO) Specification", level=1)
doc.add_paragraph(
"The platform engineering team commits to the following Service Level Objectives (SLOs) and error budgets:"
)
table = doc.add_table(rows=1, cols=4)
table.style = 'Table Grid'
hdr_cells = table.rows[0].cells
hdr_cells[0].text = "Metric / Objective"
hdr_cells[1].text = "Target SLA/SLO"
hdr_cells[2].text = "Measurement Window"
hdr_cells[3].text = "Error Budget"
slo_data = [
["API Latency (p99)", "< 120 ms", "Rolling 30 days", "0.1% requests > 120ms"],
["API Latency (p50)", "< 25 ms", "Rolling 30 days", "0.5% requests > 25ms"],
["Platform Availability", "99.99%", "Calendar Quarter", "13.14 minutes downtime"],
["MTTR (Mean Time to Recover)", "< 15 minutes", "Per P1 Incident", "Max 3 breaches / year"],
["Recovery Point Objective (RPO)", "< 1 minute", "Disaster Recovery", "Zero committed data loss"],
["Recovery Time Objective (RTO)", "< 30 minutes", "Cross-Region Failover", "Validated semi-annually"],
]
for row in slo_data:
row_cells = table.add_row().cells
for idx, text in enumerate(row):
row_cells[idx].text = text
# 5. Disaster Recovery
doc.add_heading("5. Disaster Recovery and Failover Automation", level=1)
doc.add_paragraph(
"In the event of a catastrophic regional failure in us-east-1, failover procedures are initiated through Amazon Route 53 "
"Application Recovery Controller (ARC). Health checks evaluate synthetic canary probes every 10 seconds. "
"If 3 consecutive probes fail, ARC shifts global DNS routing to us-west-2. Aurora Global Database failover is automated via "
"AWS Step Functions, promoting the secondary us-west-2 cluster to primary within 8 minutes. Total regional Recovery Time "
"Objective (RTO) is guaranteed under 30 minutes with an RPO under 60 seconds."
)
doc.save(output_path)
print(f"Generated DOCX: {output_path}")
def generate_txt(output_path: str):
content = """# Enterprise Cybersecurity & Information Governance Policy
## 1. Access Control and Identity Management (IAM)
All corporate systems and production cloud environments require mandatory multi-factor authentication (MFA) using FIDO2-compliant WebAuthn hardware security keys. SMS and time-based one-time password (TOTP) apps are prohibited for administrative and privileged infrastructure accounts.
Session timeouts are strictly enforced: inactive user console sessions automatically disconnect after 15 minutes of inactivity. Privileged command execution requires temporary elevated credentials approved via just-in-time (JIT) access approval workflows, with approvals expiring after 4 hours. All user access permissions are subject to formal quarterly access certifications reviewed by department directors.
## 2. Cryptographic and Encryption Standards
All sensitive data classified as Confidential or Restricted must be encrypted in transit and at rest:
- Data in transit: Mandatory Transport Layer Security (TLS) version 1.3. Ciphers supporting forward secrecy (ECDHE-RSA-AES128-GCM-SHA256 or better) are required. TLS 1.0, 1.1, and 1.2 are disabled on all external-facing load balancers.
- Data at rest: High-security storage volumes, S3 buckets, and relational databases must use Advanced Encryption Standard (AES) with 256-bit keys (AES-256-GCM). Cryptographic keys are managed via AWS Key Management Service (KMS) or HashiCorp Vault. All master encryption keys must undergo automated cryptographic rotation every 90 days.
## 3. Incident Response and Declaration SLA
Cybersecurity incidents are categorized into three severity levels:
- Severity 0 (P0 - Critical): Active data breach or critical infrastructure compromise. Maximum declaration SLA is 5 minutes from initial detection. An incident war room is automatically created within 10 minutes, and the Incident Commander must brief executive leadership every 30 minutes. Comprehensive post-mortem analysis must be published within 72 hours.
- Severity 1 (P1 - High): Significant system degradation or isolated security compromise without data exfiltration. Declaration SLA is 15 minutes. Post-mortem must be completed within 5 business days.
- Severity 2 (P2 - Medium): Vulnerability with potential impact or policy violation. Remediation must occur within 14 calendar days.
## 4. Vulnerability Management and Remediation Timelines
Security vulnerabilities identified through static application security testing (SAST), dynamic testing (DAST), or penetration testing must be remediated based on Common Vulnerability Scoring System (CVSS v3.1) base scores:
- Critical Vulnerabilities (CVSS 9.0 - 10.0): Remediation SLA is 48 hours from discovery.
- High Vulnerabilities (CVSS 7.0 - 8.9): Remediation SLA is 14 calendar days.
- Medium Vulnerabilities (CVSS 4.0 - 6.9): Remediation SLA is 30 calendar days.
- Low Vulnerabilities (CVSS 0.1 - 3.9): Remediation SLA is 90 calendar days or next scheduled release.
Failure to patch Critical vulnerabilities within 48 hours triggers automated quarantine of the affected host or container instance by the Security Operations Center (SOC).
## 5. Employee Endpoint and Device Security
All corporate laptops and mobile workstations must run enterprise Mobile Device Management (MDM) software with active endpoint detection and response (EDR) sensors.
Full-disk encryption (FileVault for macOS, BitLocker for Windows) is non-negotiable and enforced via MDM configuration profiles. Remote wipe capability must remain active at all times. Employees are strictly prohibited from using jailbroken, rooted, or personally owned devices (BYOD) for accessing production customer data or corporate repositories. Any lost or stolen device must be reported to security@acme.corp within 2 hours.
"""
with open(output_path, "w", encoding="utf-8") as f:
f.write(content)
print(f"Generated TXT: {output_path}")
if __name__ == "__main__":
os.makedirs("data", exist_ok=True)
generate_pdf("data/sample_financial_report.pdf")
generate_docx("data/sample_cloud_architecture.docx")
generate_txt("data/sample_security_policy.txt")