README.md

---
title: Bilingual Text Summarization NLP Suite
emoji: 📖
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 5.9.1
app_file: app.py
pinned: false
license: mit
---

# 📚 Bilingual Arabic–English NLP System for Automatic Text Summarization
### نظام المعالجة اللغوية التلقائية للتلخيص الاستخراجي والتجريدي للنصوص العربية والإنجليزية

---

## 🌟 Executive Overview
This repository contains an end-to-end **Bilingual (Arabic & English) Natural Language Processing (NLP) System for Automatic Text Summarization**, implemented completely **from scratch**.

It addresses the fundamental linguistic, structural, and morphological differences between Arabic (a morphologically rich, non-concatenative language) and English (an analytic language with fixed syntax), offering both **Extractive** and **Abstractive** summarization paradigms alongside a rigorous quantitative evaluation suite (**ROUGE**, **BLEU**, **TTR**, and **Compression Metrics**) and a modern interactive Web Dashboard.

---

## 🏛️ System Architecture

```mermaid
flowchart TD
    A[User Input: Text / PDF / DOCX / TXT] --> B[Language Detection & Character Routing]
    
    B -->|Arabic| C1[Arabic Preprocessing Engine: Tashkeel, Normalization, ISRI Stemmer]
    B -->|English| C2[English Preprocessing Engine: Contractions, Lemmatizer, Filter]
    
    C1 --> D{Summarization Mode}
    C2 --> D
    
    D -->|Extractive| E1[Sentence Segmentation & TF-IDF Vectorizer]
    E1 --> E2[Graph-based TextRank / LSA SVD / Hybrid Centroid]
    E2 --> E3[Extractive Chronological Summary]
    
    D -->|Abstractive| F1[Bilingual Vocabulary & Dynamic Pad Batching]
    F1 --> F2[PyTorch Seq2Seq with Bahdanau / Luong Attention]
    F2 --> F3[Beam Search / Greedy Decoding]
    F3 --> F4[Abstractive Paraphrased Summary]
    
    E3 --> G[Evaluation Suite: ROUGE-1/2/L, BLEU-1..4, Diversity & Ratio]
    F4 --> G
    
    G --> H[Streamlit Interactive Web GUI & CLI]
```

---

## 📁 Repository Structure

```text
sumary project/
│
├── data/
│   ├── sample_arabic.txt        # Comprehensive Arabic sample text
│   ├── sample_english.txt       # Comprehensive English sample text
│   ├── arabic_stopwords.txt     # Curated Arabic stopword lexicon
│   ├── english_stopwords.txt    # Standard English stopword lexicon
│   └── datasets/                # Generated bilingual parallel corpora (.json)
│
├── nlp_core/                    # Fundamental NLP and preprocessing engine
│   ├── __init__.py
│   ├── language_detector.py     # Unicode & n-gram language classifier (AR/EN)
│   ├── arabic_preprocessor.py   # Tashkeel removal, Alef/Yaa/Taa normalization, Light stemmer
│   ├── english_preprocessor.py  # Contraction expansion, lowercasing, stemmer
│   ├── tokenizer.py             # Arabic & English sentence segmenter and word tokenizer
│   └── vocabulary.py            # Vocab manager, token encoder/decoder, special tokens
│
├── models/
│   ├── extractive/              # Extractive summarization algorithms (From Scratch)
│   │   ├── tfidf.py             # Custom Sublinear TF-IDF & Cosine Similarity
│   │   ├── textrank.py          # Graph PageRank power iteration summarizer
│   │   ├── lsa.py               # Latent Semantic Analysis with SVD
│   │   └── hybrid_scorer.py     # Multi-feature (TextRank + Centroid + Position + Length)
│   │
│   └── abstractive/             # Deep learning Sequence-to-Sequence (PyTorch From Scratch)
│       ├── attention.py         # Bahdanau (Additive) & Luong Attention mechanisms
│       ├── encoder.py           # Bidirectional GRU/LSTM Encoder
│       ├── decoder.py           # Attention-guided GRU Decoder
│       ├── seq2seq_model.py     # Full Seq2Seq model with Greedy & Beam Search
│       └── transformer_wrap.py  # Modular wrapper for Hugging Face Pretrained Models
│
├── training/                    # ML/DL Training and Dataset pipeline
│   ├── sample_data_gen.py       # Bilingual parallel dataset synthesizer
│   ├── dataset.py               # PyTorch Dataset and dynamic pad collator
│   └── train.py                 # Training loop with teacher forcing, validation, checkpointing
│
├── evaluation/                  # Quantitative benchmark suite (From Scratch)
│   ├── rouge.py                 # ROUGE-1, ROUGE-2, and ROUGE-L (Precision, Recall, F1)
│   ├── bleu.py                  # BLEU-1 to BLEU-4 and Cumulative BLEU with Brevity Penalty
│   └── metrics_manager.py       # Unified evaluator (Compression, Diversity, Overlap)
│
├── checkpoints/                 # Saved PyTorch model weights (.pt)
│   ├── seq2seq_arabic.pt
│   └── seq2seq_english.pt
│
├── tests/                       # Complete automated unit test suite
│   ├── test_preprocessing.py
│   ├── test_extractive.py
│   ├── test_abstractive.py
│   └── test_evaluation.py
│
├── app.py                       # Modern Interactive Streamlit Web GUI
├── main_cli.py                  # Versatile Command-Line Interface
├── requirements.txt             # Python dependencies
└── README.md                    # Documentation & System Manual
```

---

## 🚀 Getting Started

### 1. Installation & Environment Setup

```bash
# Clone the repository and navigate to folder
cd "sumary project"

# Install dependencies
pip install -r requirements.txt
```

### 2. Run Automated Unit Tests

```bash
python -m unittest discover tests
```
*Executes all 14 unit tests validating preprocessing, TF-IDF, TextRank, LSA, Seq2Seq Attention forward pass, ROUGE, and BLEU.*

---

## 💻 Usage

### 1. Interactive Web Application (GUI)

Launch the modern glassmorphic dashboard:
```bash
streamlit run app.py
```

**Features in the Web App:**
- 📄 Upload `.txt`, `.pdf`, or `.docx` documents.
- 🌐 Automatic language detection for Arabic and English.
- 🎛️ Toggle between **Extractive** (TextRank, LSA, Hybrid) and **Abstractive** (Seq2Seq Attention, Transformers).
- 🔍 Visual sentence highlighting showing exact extracted sentences in the source text.
- 📈 Live quantitative ROUGE and BLEU calculation against reference summaries.
- 🏋️ **Training Studio tab** to train and fine-tune Seq2Seq models directly from the GUI.
- 💾 One-click export to download generated summaries.

---

### 2. Command-Line Interface (CLI)

#### A. Arabic Extractive Summarization (TextRank)
```bash
python main_cli.py --file data/sample_arabic.txt --mode extractive --method textrank --sentences 2
```

#### B. English Extractive Summarization (Hybrid Multi-Feature)
```bash
python main_cli.py --file data/sample_english.txt --mode extractive --method hybrid --sentences 3
```

#### C. Arabic Abstractive Summarization (Neural Seq2Seq with Attention)
```bash
python main_cli.py --text "تعد معالجة اللغات الطبيعية فرعا جوهريا من الذكاء الاصطناعي لفهم وتوليد النصوص البشرية." --mode abstractive --method seq2seq --beam 3
```

#### D. Summarize and Evaluate with ROUGE & BLEU Benchmarks
```bash
python main_cli.py --file data/sample_english.txt --mode extractive --method textrank --ref "Natural language processing enables computers to comprehend text and summarize documents efficiently."
```

#### E. JSON Output for API / Automated Pipelines
```bash
python main_cli.py --file data/sample_arabic.txt --mode extractive --json
```

---

## 🧠 Model Training

To retrain the neural Seq2Seq models on custom or augmented datasets:

```bash
# Train both Arabic and English models for 10 epochs
python training/train.py --lang all --epochs 10 --batch_size 8 --lr 0.002

# Train only Arabic model
python training/train.py --lang ar --epochs 15

# Train only English model
python training/train.py --lang en --epochs 15
```

Checkpoints are automatically saved to `checkpoints/seq2seq_arabic.pt` and `checkpoints/seq2seq_english.pt`.

---

## 📊 Evaluation & Mathematical Formulations

### 1. ROUGE-N (Recall, Precision, F1)
$$\text{ROUGE-N Recall} = \frac{\sum_{S \in \{\text{Reference}\}} \sum_{\text{gram}_n \in S} \text{Count}_{\text{match}}(\text{gram}_n)}{\sum_{S \in \{\text{Reference}\}} \sum_{\text{gram}_n \in S} \text{Count}(\text{gram}_n)}$$

### 2. BLEU Score with Brevity Penalty (BP)
$$\text{BLEU} = \text{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \ln p_n\right), \quad \text{where } \text{BP} = \begin{cases} 1 & \text{if } c > r \\ e^{(1 - r/c)} & \text{if } c \le r \end{cases}$$

### 3. PageRank Centrality (TextRank)
$$WS(V_i) = (1 - d) + d \sum_{V_j \in \text{In}(V_i)} \frac{w_{ji}}{\sum_{V_k \in \text{Out}(V_j)} w_{jk}} WS(V_j)$$

### 4. Bahdanau Additive Attention
$$e_{ij} = v_a^T \tanh(W_a s_{i-1} + U_a h_j), \quad \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}, \quad c_i = \sum_j \alpha_{ij} h_j$$

---

## 📜 License
This project is licensed under the MIT License - open for educational, academic research, and commercial use.