Hugging Face's logo

AALAM-Studio
/
warraqa

Model card Files
xet
 Community
warraqa / README.md
alhajjaj's picture
alhajjaj
Upload README.md with huggingface_hub
2ef40cb verified
preview code
|
Raw History Blame Contribute Delete
8.54 kB

license: polyform-noncommercial-1.0.0 title: Warraqa sdk: python

Warraqa (ΩˆΨ±Ω‘Ψ§Ω‚Ψ©)

The Document Scribe Agent

Named after the WarrāqΕ«n β€” the master scribes and paper-makers of the Islamic Golden Age.

License: PolyForm NC 1.0.0 Python 3.10+ Version CI Aalam Studio

Warraqa converts PDF, Word, and PowerPoint documents into clean, accurate Markdown β€” and scores her own work.

Why Warraqa?

Most document-to-Markdown tools are one-trick ponies: great at clean PDFs, terrible at scans; great at .docx, blind to .doc; or they silently produce garbage and let you discover it three pipelines later.

Warraqa is a specialist agent. She picks the right engine for each file, falls back gracefully, scores her output from 0–100 with letter grades, and tells you which conversions to trust. She's built to feed RAG pipelines, knowledge bases, and downstream agents β€” where Markdown quality directly determines retrieval quality.

Features

  • Dual-engine architecture β€” best specialized tool for each format
    • Marker (deep learning) for scanned PDFs: tables, equations, multi-column, OCR
    • PyMuPDF4LLM (fast, CPU-only) for native-text PDFs
    • MarkItDown (Microsoft) for .docx and .pptx
    • MS Office COM auto-converts legacy .doc and .ppt to modern formats first
    • Pandoc fallback for .docx resilience
  • Smart triage β€” every PDF is pre-scanned to detect native vs. scanned content; routing is automatic
  • Two-phase batch processing β€” fast files (native PDFs, Word, PowerPoint) run first; slow OCR work is deferred to a single trailing pass so you don't wait on Marker mid-batch
  • Quality scoring β€” every conversion gets a 0–100 confidence score with an A–F grade across 5 dimensions (completeness, structure, encoding, density, readability)
  • Crash-resistant β€” sanitizes invalid Unicode from upstream engines so a single bad PDF can't kill a 1000-file run
  • Folder workflow β€” input β†’ convert β†’ output + move originals to processed/ or failed/
  • Watch mode β€” continuous monitoring for new files
  • Inter-agent API β€” designed for other agents to call programmatically

Quick Start

Option 1 β€” pip (recommended) 

pip install warraqa
warraqa --folder "C:\path\to\documents"

You still need Pandoc on PATH for .docx fallback, and MS Office (Windows) for legacy .doc/.ppt. The Marker engine downloads its ML models on first use (~2–3 GB).

Option 2 β€” Clone + bootstrap script 

git clone https://github.com/AALAM-Studio/warraqa.git
cd warraqa
python bootstrap.py        # creates .venv, installs deps, auto-installs Pandoc on Windows
.venv\Scripts\activate     # Linux/macOS: source .venv/bin/activate
python run.py

Option 3 β€” Docker (for cloud / headless use) 

docker build -t warraqa .
docker run --rm -v "/path/to/docs:/data" warraqa --folder /data

Note: the Docker image is CPU-only and does not include MS Office, so legacy .doc/.ppt will be skipped with a clean error message.

Usage 

warraqa                              # Manual mode β€” opens a folder picker dialog
warraqa --folder "C:\path"           # Process a specific folder
warraqa --file path/to/document.pdf  # Convert a single file
warraqa --watch --folder "C:\path"   # Watch mode β€” continuously monitor
warraqa --folder "C:\path" --no-save --no-move    # Dry run
warraqa --help                       # All options

Output Structure 

output/
β”œβ”€β”€ md_files/        # Converted Markdown files
β”œβ”€β”€ processed/       # Successfully converted originals
β”œβ”€β”€ failed/          # Failed conversion originals
β”œβ”€β”€ reports/         # JSON reports with scores and metadata
β”œβ”€β”€ scanned_pdfs/    # Staging area for OCR-bound PDFs (auto-cleaned per run)
└── warraqa.log

Quality Scoring

Every conversion is scored across 5 weighted dimensions:

Dimension Weight What It Measures
Text Completeness 30% Word count vs. expected density for file size
Structure Integrity 25% Headings, lists, tables, formatting
Encoding Quality 20% Garbled text, mojibake, Unicode issues
Content Density 15% Meaningful text vs. noise
Readability 10% Line length, paragraph structure

Grades: A (90–100) β†’ B (75–89) β†’ C (60–74) β†’ D (40–59) β†’ F (0–39). Files scoring below 40 are moved to output/failed/ automatically.

Inter-Agent API 

from warraqa import Warraqa

agent = Warraqa()

# Convert a single file
result = agent.convert_file("document.pdf")
print(result.confidence_score)    # 87
print(result.grade)               # "B"
print(result.markdown_content)    # "# Title\n\n..."
print(result.output_path)         # Path to saved .md file

# Process a folder
results = agent.process_folder("C:/Users/you/Academia")
for r in results:
    print(f"{r.source_file.filename}: {r.grade} ({r.confidence_score}/100)")

Configuration

Edit config.yaml to customize:

  • Default mode (manual / watch)
  • Output directories
  • Engine preferences (primary / fallback per format)
  • Scoring thresholds
  • Logging level

Supported Formats

Extension Engine Notes
.pdf (native text) PyMuPDF4LLM Fast, CPU-only
.pdf (scanned) Marker Deferred to Phase 2 OCR pass
.docx MarkItDown β†’ Pandoc β€”
.doc MS Office COM β†’ MarkItDown Windows + Office required
.pptx MarkItDown β€”
.ppt MS Office COM β†’ MarkItDown Windows + Office required

License

Warraqa is published under the PolyForm Noncommercial License 1.0.0 β€” a source-available license that allows free use for:

  • Personal projects, research, study, and experimentation
  • Academic and educational institutions
  • Charitable, public-safety, health, and government organizations
  • Internal evaluation by any organization

Commercial use β€” including using Warraqa as part of a product or service offered to paying customers, internal business operations at a for-profit company, or any revenue-generating workflow β€” requires a separate commercial license. Contact contact@aalam.consulting to discuss licensing.

Note on terminology: PolyForm Noncommercial is source-available, not open source in the OSI sense (which by definition allows commercial use). The full text is in LICENSE.

Versioning Policy

This repository contains Warraqa v1.0.0 β€” the inaugural public, source-available release. Future versions of Warraqa are developed privately and available under commercial license terms. Critical bug fixes may be backported to v1.x at AALAM Studio's discretion.

See CHANGELOG.md for the release history.

Citation

If Warraqa contributes to academic research, please cite it. A machine-readable CITATION.cff is provided, or use the GitHub "Cite this repository" button.

Acknowledgements

Warraqa stands on the shoulders of excellent open-source projects:

  • Marker β€” Vik Paruchuri's deep-learning PDF parser
  • PyMuPDF4LLM β€” Artifex's LLM-optimized PDF extraction
  • MarkItDown β€” Microsoft's universal-to-markdown converter
  • Pandoc β€” John MacFarlane's document conversion swiss-army knife
  • Rich β€” Will McGugan's terminal beautifier

Part of Aalam Studio

Warraqa is the first publicly released agent in the AALAM Studio ecosystem. Other agents access her output at a predictable path:

WARRAQA_OUTPUT = "c:/projects/aalam-studio/warraqa/output/"

She reads. She transcribes. She scores her own work.

Built with care by AALAM Studio.