| <div align="center"> |
|
|
| --- |
| license: polyform-noncommercial-1.0.0 |
| title: Warraqa |
| sdk: python |
| --- |
|
|
| # Warraqa (ΩΨ±ΩΨ§ΩΨ©) |
|
|
| ### The Document Scribe Agent |
|
|
| *Named after the **WarrΔqΕ«n** β the master scribes and paper-makers of the Islamic Golden Age.* |
|
|
| [](LICENSE) |
| [](https://www.python.org/) |
| [](CHANGELOG.md) |
| [](https://github.com/AALAM-Studio/warraqa/actions/workflows/ci.yml) |
| [](https://github.com/AALAM-Studio) |
|
|
| **Warraqa converts PDF, Word, and PowerPoint documents into clean, accurate Markdown β and scores her own work.** |
|
|
| </div> |
|
|
| --- |
|
|
| ## Why Warraqa? |
|
|
| Most document-to-Markdown tools are one-trick ponies: great at clean PDFs, terrible at scans; great at `.docx`, blind to `.doc`; or they silently produce garbage and let you discover it three pipelines later. |
|
|
| Warraqa is a **specialist agent**. She picks the right engine for each file, falls back gracefully, scores her output from 0β100 with letter grades, and tells you which conversions to trust. She's built to feed RAG pipelines, knowledge bases, and downstream agents β where Markdown quality directly determines retrieval quality. |
|
|
| ## Features |
|
|
| - **Dual-engine architecture** β best specialized tool for each format |
| - **Marker** (deep learning) for scanned PDFs: tables, equations, multi-column, OCR |
| - **PyMuPDF4LLM** (fast, CPU-only) for native-text PDFs |
| - **MarkItDown** (Microsoft) for `.docx` and `.pptx` |
| - **MS Office COM** auto-converts legacy `.doc` and `.ppt` to modern formats first |
| - Pandoc fallback for `.docx` resilience |
| - **Smart triage** β every PDF is pre-scanned to detect native vs. scanned content; routing is automatic |
| - **Two-phase batch processing** β fast files (native PDFs, Word, PowerPoint) run first; slow OCR work is deferred to a single trailing pass so you don't wait on Marker mid-batch |
| - **Quality scoring** β every conversion gets a 0β100 confidence score with an AβF grade across 5 dimensions (completeness, structure, encoding, density, readability) |
| - **Crash-resistant** β sanitizes invalid Unicode from upstream engines so a single bad PDF can't kill a 1000-file run |
| - **Folder workflow** β input β convert β output + move originals to `processed/` or `failed/` |
| - **Watch mode** β continuous monitoring for new files |
| - **Inter-agent API** β designed for other agents to call programmatically |
|
|
| ## Quick Start |
|
|
| ### Option 1 β pip (recommended) |
|
|
| ```bash |
| pip install warraqa |
| warraqa --folder "C:\path\to\documents" |
| ``` |
|
|
| You still need **Pandoc** on `PATH` for `.docx` fallback, and **MS Office** (Windows) for legacy `.doc`/`.ppt`. The Marker engine downloads its ML models on first use (~2β3 GB). |
|
|
| ### Option 2 β Clone + bootstrap script |
|
|
| ```bash |
| git clone https://github.com/AALAM-Studio/warraqa.git |
| cd warraqa |
| python bootstrap.py # creates .venv, installs deps, auto-installs Pandoc on Windows |
| .venv\Scripts\activate # Linux/macOS: source .venv/bin/activate |
| python run.py |
| ``` |
|
|
| ### Option 3 β Docker (for cloud / headless use) |
|
|
| ```bash |
| docker build -t warraqa . |
| docker run --rm -v "/path/to/docs:/data" warraqa --folder /data |
| ``` |
|
|
| Note: the Docker image is CPU-only and does **not** include MS Office, so legacy `.doc`/`.ppt` will be skipped with a clean error message. |
|
|
| ## Usage |
|
|
| ```bash |
| warraqa # Manual mode β opens a folder picker dialog |
| warraqa --folder "C:\path" # Process a specific folder |
| warraqa --file path/to/document.pdf # Convert a single file |
| warraqa --watch --folder "C:\path" # Watch mode β continuously monitor |
| warraqa --folder "C:\path" --no-save --no-move # Dry run |
| warraqa --help # All options |
| ``` |
|
|
| ## Output Structure |
|
|
| ``` |
| output/ |
| βββ md_files/ # Converted Markdown files |
| βββ processed/ # Successfully converted originals |
| βββ failed/ # Failed conversion originals |
| βββ reports/ # JSON reports with scores and metadata |
| βββ scanned_pdfs/ # Staging area for OCR-bound PDFs (auto-cleaned per run) |
| βββ warraqa.log |
| ``` |
|
|
| ## Quality Scoring |
|
|
| Every conversion is scored across 5 weighted dimensions: |
|
|
| | Dimension | Weight | What It Measures | |
| |:--|:--|:--| |
| | Text Completeness | 30% | Word count vs. expected density for file size | |
| | Structure Integrity | 25% | Headings, lists, tables, formatting | |
| | Encoding Quality | 20% | Garbled text, mojibake, Unicode issues | |
| | Content Density | 15% | Meaningful text vs. noise | |
| | Readability | 10% | Line length, paragraph structure | |
|
|
| Grades: **A** (90β100) β **B** (75β89) β **C** (60β74) β **D** (40β59) β **F** (0β39). |
| Files scoring below 40 are moved to `output/failed/` automatically. |
|
|
| ## Inter-Agent API |
|
|
| ```python |
| from warraqa import Warraqa |
| |
| agent = Warraqa() |
| |
| # Convert a single file |
| result = agent.convert_file("document.pdf") |
| print(result.confidence_score) # 87 |
| print(result.grade) # "B" |
| print(result.markdown_content) # "# Title\n\n..." |
| print(result.output_path) # Path to saved .md file |
| |
| # Process a folder |
| results = agent.process_folder("C:/Users/you/Academia") |
| for r in results: |
| print(f"{r.source_file.filename}: {r.grade} ({r.confidence_score}/100)") |
| ``` |
|
|
| ## Configuration |
|
|
| Edit [`config.yaml`](config.yaml) to customize: |
|
|
| - Default mode (manual / watch) |
| - Output directories |
| - Engine preferences (primary / fallback per format) |
| - Scoring thresholds |
| - Logging level |
|
|
| ## Supported Formats |
|
|
| | Extension | Engine | Notes | |
| |:--|:--|:--| |
| | `.pdf` (native text) | PyMuPDF4LLM | Fast, CPU-only | |
| | `.pdf` (scanned) | Marker | Deferred to Phase 2 OCR pass | |
| | `.docx` | MarkItDown β Pandoc | β | |
| | `.doc` | MS Office COM β MarkItDown | Windows + Office required | |
| | `.pptx` | MarkItDown | β | |
| | `.ppt` | MS Office COM β MarkItDown | Windows + Office required | |
|
|
| ## License |
|
|
| **Warraqa is published under the [PolyForm Noncommercial License 1.0.0](LICENSE)** β a source-available license that allows free use for: |
|
|
| - Personal projects, research, study, and experimentation |
| - Academic and educational institutions |
| - Charitable, public-safety, health, and government organizations |
| - Internal evaluation by any organization |
|
|
| **Commercial use** β including using Warraqa as part of a product or service offered to paying customers, internal business operations at a for-profit company, or any revenue-generating workflow β **requires a separate commercial license**. Contact **`contact@aalam.consulting`** to discuss licensing. |
|
|
| Note on terminology: PolyForm Noncommercial is *source-available*, not *open source* in the OSI sense (which by definition allows commercial use). The full text is in [`LICENSE`](LICENSE). |
|
|
| ## Versioning Policy |
|
|
| This repository contains **Warraqa v1.0.0** β the inaugural public, source-available release. Future versions of Warraqa are developed privately and available under commercial license terms. Critical bug fixes may be backported to v1.x at AALAM Studio's discretion. |
|
|
| See [`CHANGELOG.md`](CHANGELOG.md) for the release history. |
|
|
| ## Citation |
|
|
| If Warraqa contributes to academic research, please cite it. A machine-readable [`CITATION.cff`](CITATION.cff) is provided, or use the GitHub "Cite this repository" button. |
|
|
| ## Acknowledgements |
|
|
| Warraqa stands on the shoulders of excellent open-source projects: |
|
|
| - [Marker](https://github.com/VikParuchuri/marker) β Vik Paruchuri's deep-learning PDF parser |
| - [PyMuPDF4LLM](https://github.com/pymupdf/PyMuPDF) β Artifex's LLM-optimized PDF extraction |
| - [MarkItDown](https://github.com/microsoft/markitdown) β Microsoft's universal-to-markdown converter |
| - [Pandoc](https://pandoc.org/) β John MacFarlane's document conversion swiss-army knife |
| - [Rich](https://github.com/Textualize/rich) β Will McGugan's terminal beautifier |
|
|
| ## Part of Aalam Studio |
|
|
| Warraqa is the first publicly released agent in the **AALAM Studio** ecosystem. Other agents access her output at a predictable path: |
|
|
| ```python |
| WARRAQA_OUTPUT = "c:/projects/aalam-studio/warraqa/output/" |
| ``` |
|
|
| --- |
|
|
| <div align="center"> |
|
|
| *She reads. She transcribes. She scores her own work.* |
|
|
| **Built with care by [AALAM Studio](https://github.com/AALAM-Studio).** |
|
|
| </div> |
|
|
