README.md

metadata

title: Bilingual Text Summarization NLP Suite
emoji: 📖
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 5.9.1
app_file: app.py
pinned: false
license: mit

📚 Bilingual Arabic–English NLP System for Automatic Text Summarization

نظام المعالجة اللغوية التلقائية للتلخيص الاستخراجي والتجريدي للنصوص العربية والإنجليزية

---

🌟 Executive Overview

This repository contains an end-to-end Bilingual (Arabic & English) Natural Language Processing (NLP) System for Automatic Text Summarization, implemented completely from scratch.

It addresses the fundamental linguistic, structural, and morphological differences between Arabic (a morphologically rich, non-concatenative language) and English (an analytic language with fixed syntax), offering both Extractive and Abstractive summarization paradigms alongside a rigorous quantitative evaluation suite (ROUGE, BLEU, TTR, and Compression Metrics) and a modern interactive Web Dashboard.

---

🏛️ System Architecture

flowchart TD
    A[User Input: Text / PDF / DOCX / TXT] --> B[Language Detection & Character Routing]
    
    B -->|Arabic| C1[Arabic Preprocessing Engine: Tashkeel, Normalization, ISRI Stemmer]
    B -->|English| C2[English Preprocessing Engine: Contractions, Lemmatizer, Filter]
    
    C1 --> D{Summarization Mode}
    C2 --> D
    
    D -->|Extractive| E1[Sentence Segmentation & TF-IDF Vectorizer]
    E1 --> E2[Graph-based TextRank / LSA SVD / Hybrid Centroid]
    E2 --> E3[Extractive Chronological Summary]
    
    D -->|Abstractive| F1[Bilingual Vocabulary & Dynamic Pad Batching]
    F1 --> F2[PyTorch Seq2Seq with Bahdanau / Luong Attention]
    F2 --> F3[Beam Search / Greedy Decoding]
    F3 --> F4[Abstractive Paraphrased Summary]
    
    E3 --> G[Evaluation Suite: ROUGE-1/2/L, BLEU-1..4, Diversity & Ratio]
    F4 --> G
    
    G --> H[Streamlit Interactive Web GUI & CLI]

---

📁 Repository Structure

sumary project/
│
├── data/
│   ├── sample_arabic.txt        # Comprehensive Arabic sample text
│   ├── sample_english.txt       # Comprehensive English sample text
│   ├── arabic_stopwords.txt     # Curated Arabic stopword lexicon
│   ├── english_stopwords.txt    # Standard English stopword lexicon
│   └── datasets/                # Generated bilingual parallel corpora (.json)
│
├── nlp_core/                    # Fundamental NLP and preprocessing engine
│   ├── __init__.py
│   ├── language_detector.py     # Unicode & n-gram language classifier (AR/EN)
│   ├── arabic_preprocessor.py   # Tashkeel removal, Alef/Yaa/Taa normalization, Light stemmer
│   ├── english_preprocessor.py  # Contraction expansion, lowercasing, stemmer
│   ├── tokenizer.py             # Arabic & English sentence segmenter and word tokenizer
│   └── vocabulary.py            # Vocab manager, token encoder/decoder, special tokens
│
├── models/
│   ├── extractive/              # Extractive summarization algorithms (From Scratch)
│   │   ├── tfidf.py             # Custom Sublinear TF-IDF & Cosine Similarity
│   │   ├── textrank.py          # Graph PageRank power iteration summarizer
│   │   ├── lsa.py               # Latent Semantic Analysis with SVD
│   │   └── hybrid_scorer.py     # Multi-feature (TextRank + Centroid + Position + Length)
│   │
│   └── abstractive/             # Deep learning Sequence-to-Sequence (PyTorch From Scratch)
│       ├── attention.py         # Bahdanau (Additive) & Luong Attention mechanisms
│       ├── encoder.py           # Bidirectional GRU/LSTM Encoder
│       ├── decoder.py           # Attention-guided GRU Decoder
│       ├── seq2seq_model.py     # Full Seq2Seq model with Greedy & Beam Search
│       └── transformer_wrap.py  # Modular wrapper for Hugging Face Pretrained Models
│
├── training/                    # ML/DL Training and Dataset pipeline
│   ├── sample_data_gen.py       # Bilingual parallel dataset synthesizer
│   ├── dataset.py               # PyTorch Dataset and dynamic pad collator
│   └── train.py                 # Training loop with teacher forcing, validation, checkpointing
│
├── evaluation/                  # Quantitative benchmark suite (From Scratch)
│   ├── rouge.py                 # ROUGE-1, ROUGE-2, and ROUGE-L (Precision, Recall, F1)
│   ├── bleu.py                  # BLEU-1 to BLEU-4 and Cumulative BLEU with Brevity Penalty
│   └── metrics_manager.py       # Unified evaluator (Compression, Diversity, Overlap)
│
├── checkpoints/                 # Saved PyTorch model weights (.pt)
│   ├── seq2seq_arabic.pt
│   └── seq2seq_english.pt
│
├── tests/                       # Complete automated unit test suite
│   ├── test_preprocessing.py
│   ├── test_extractive.py
│   ├── test_abstractive.py
│   └── test_evaluation.py
│
├── app.py                       # Modern Interactive Streamlit Web GUI
├── main_cli.py                  # Versatile Command-Line Interface
├── requirements.txt             # Python dependencies
└── README.md                    # Documentation & System Manual

---

🚀 Getting Started

1. Installation & Environment Setup

# Clone the repository and navigate to folder
cd "sumary project"

# Install dependencies
pip install -r requirements.txt

2. Run Automated Unit Tests

python -m unittest discover tests

Executes all 14 unit tests validating preprocessing, TF-IDF, TextRank, LSA, Seq2Seq Attention forward pass, ROUGE, and BLEU.

---

💻 Usage

1. Interactive Web Application (GUI)

Launch the modern glassmorphic dashboard:

streamlit run app.py

Features in the Web App:

• 📄 Upload .txt, .pdf, or .docx documents.
• 🌐 Automatic language detection for Arabic and English.
• 🎛️ Toggle between Extractive (TextRank, LSA, Hybrid) and Abstractive (Seq2Seq Attention, Transformers).
• 🔍 Visual sentence highlighting showing exact extracted sentences in the source text.
• 📈 Live quantitative ROUGE and BLEU calculation against reference summaries.
• 🏋️ Training Studio tab to train and fine-tune Seq2Seq models directly from the GUI.
• 💾 One-click export to download generated summaries.

---

2. Command-Line Interface (CLI)

A. Arabic Extractive Summarization (TextRank)

python main_cli.py --file data/sample_arabic.txt --mode extractive --method textrank --sentences 2

B. English Extractive Summarization (Hybrid Multi-Feature)

python main_cli.py --file data/sample_english.txt --mode extractive --method hybrid --sentences 3

C. Arabic Abstractive Summarization (Neural Seq2Seq with Attention)

python main_cli.py --text "تعد معالجة اللغات الطبيعية فرعا جوهريا من الذكاء الاصطناعي لفهم وتوليد النصوص البشرية." --mode abstractive --method seq2seq --beam 3

D. Summarize and Evaluate with ROUGE & BLEU Benchmarks

python main_cli.py --file data/sample_english.txt --mode extractive --method textrank --ref "Natural language processing enables computers to comprehend text and summarize documents efficiently."

E. JSON Output for API / Automated Pipelines

python main_cli.py --file data/sample_arabic.txt --mode extractive --json

---

🧠 Model Training

To retrain the neural Seq2Seq models on custom or augmented datasets:

# Train both Arabic and English models for 10 epochs
python training/train.py --lang all --epochs 10 --batch_size 8 --lr 0.002

# Train only Arabic model
python training/train.py --lang ar --epochs 15

# Train only English model
python training/train.py --lang en --epochs 15

Checkpoints are automatically saved to checkpoints/seq2seq_arabic.pt and checkpoints/seq2seq_english.pt.

---

📊 Evaluation & Mathematical Formulations

1. ROUGE-N (Recall, Precision, F1)

$$\text{ROUGE-N Recall}=\frac{\sum _{S\in \{\text{Reference}\}}\sum _{{\text{gram}}_n\in S}{\text{Count}}_{\text{match}}({\text{gram}}_n)}{\sum _{S\in \{\text{Reference}\}}\sum _{{\text{gram}}_n\in S}\text{Count}({\text{gram}}_n)}\backslash text\{ROUGE-N\; Recall\}\; =\; \backslash frac\{\backslash sum\_\{S\; \backslash in\; \backslash \{\backslash text\{Reference\}\backslash \}\}\; \backslash sum\_\{\backslash text\{gram\}\_n\; \backslash in\; S\}\; \backslash text\{Count\}\_\{\backslash text\{match\}\}(\backslash text\{gram\}\_n)\}\{\backslash sum\_\{S\; \backslash in\; \backslash \{\backslash text\{Reference\}\backslash \}\}\; \backslash sum\_\{\backslash text\{gram\}\_n\; \backslash in\; S\}\; \backslash text\{Count\}(\backslash text\{gram\}\_n)\}$$

2. BLEU Score with Brevity Penalty (BP)

3. PageRank Centrality (TextRank)

$$WS(V_i)=(1-d)+d\sum _{V_j\in \text{In}(V_i)}\frac{w_{ji}}{\sum _{V_k\in \text{Out}(V_j)}{w}_{jk}}WS(V_j)WS(V\_i)\; =\; (1\; -\; d)\; +\; d\; \backslash sum\_\{V\_j\; \backslash in\; \backslash text\{In\}(V\_i)\}\; \backslash frac\{w\_\{ji\}\}\{\backslash sum\_\{V\_k\; \backslash in\; \backslash text\{Out\}(V\_j)\}\; w\_\{jk\}\}\; WS(V\_j)$$

4. Bahdanau Additive Attention

$$e_{ij}=v_a^T\tanh (W_a{s}_{i-1}+U_a{h}_j),{\alpha }_{ij}=\frac{\exp (e_{ij})}{\sum _k\exp (e_{ik})},c_i=\sum _j{\alpha }_{ij}{h}_{j}e\_\{ij\}\; =\; v\_a^T\; \backslash tanh(W\_a\; s\_\{i-1\}\; +\; U\_a\; h\_j),\; \backslash quad\; \backslash alpha\_\{ij\}\; =\; \backslash frac\{\backslash exp(e\_\{ij\})\}\{\backslash sum\_k\; \backslash exp(e\_\{ik\})\},\; \backslash quad\; c\_i\; =\; \backslash sum\_j\; \backslash alpha\_\{ij\}\; h\_j$$

---

📜 License

This project is licensed under the MIT License - open for educational, academic research, and commercial use.