Email-Segregation / README.md
archittmittal's picture
Update README.md
8cac6f3 unverified
|
Raw History Blame Contribute Delete
6.7 kB
---
title: Email Segregation
emoji: πŸ“§
colorFrom: blue
colorTo: indigo
sdk: docker
python_version: "3.9"
app_file: app.py
pinned: false
---
# βš“ ShipSeg: Automated Shipping Email Segregation & Data Extraction System
[![Vercel App](https://img.shields.io/badge/%E2%96%B2%20Vercel-Deployed-black)](https://email-segregation.vercel.app/)
[![Hugging Face Backend](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Backend-blue)](https://huggingface.co/spaces/archittmittal/Email-Segregation)
ShipSeg is a modular, high-performance local pipeline built to ingest, classify, and extract commercial data from unstructured shipping broker emails and attachments.
By automating the transition from messy emails to a structured database, the platform eliminates manual data entry, reduces vessel-to-cargo matching latency, and increases market visibility.
---
## πŸ“‹ The Business Problem
Maritime chartering desks and shipbrokers receive thousands of emails every day containing raw, unstandardized lists of **vessel availability (Tonnage)** and **cargo requirements (Voyage/Time Charters)**. Manually sorting these and typing them into a database leads to:
* **Missed Opportunities**: High-value matches go unnoticed in overloaded inboxes.
* **Operational Bottlenecks**: Significant latency in response time to broker offers.
* **Data Inaccuracy**: Human error in capturing critical values (laycan dates, draft sizes, ports).
---
## ⚑ System Constraints & Architecture
> [!IMPORTANT]
> **Zero Third-Party LLM Dependency Constraint**: To guarantee absolute data privacy, eliminate API subscription costs, and ensure instant offline latency, this system runs **entirely local machine learning models and heuristics**. It does **NOT** query OpenAI, Anthropic, or any external AI services.
### Technical Stack
* **Backend**: Python 3.9+ / Flask (REST APIs & static routing)
* **Local ML & NLP**: Scikit-Learn (TF-IDF Vectorization + Multinomial Logistic Regression)
* **Data Layer**: SQLAlchemy ORM / SQLite
* **Frontend**: Responsive Single Page Application (SPA) built using Vanilla JavaScript, HTML5, and a custom CSS premium dark-mode theme.
---
## πŸ”„ Pipeline Workflow
```mermaid
flowchart TD
A[Raw Incoming Email / Attachment] --> B[Ingestion Layer]
B -->|Parse EML, TXT, or PDF| C[Deduplication Layer]
C -->|SHA-256 Fingerprint Check| D{Is Duplicate?}
D -->|Yes| E[Flag as Duplicate & Link Source]
D -->|No| F[Classification Engine]
F -->|Local ML / TF-IDF + Logistic Reg| G{Confidence > 55%?}
G -->|No| H[Regex Rule Fallback Classifier]
G -->|Yes| I[Category Route Matcher]
H --> I
I -->|Tonnage| J[Tonnage Extractor]
I -->|Cargo VC| K[Cargo VC Extractor]
I -->|Cargo TC| L[Cargo TC Extractor]
J & K & L --> M[Database Persistence]
M --> N[Matching Engine]
N --> O[Ranked Vessel-Cargo Matching Score]
```
---
## Core Module Specifications
### 1. Ingestion & Normalization (`ingestion/`)
* **Multi-Format Support**: Parses raw email text paste blocks, standard `.eml` email files, `.txt` files, and `.pdf` attachments.
* **Fallback PDF Parser**: Attempts extraction via `pdfminer.six`. If missing locally, it falls back to a custom binary regex stream scanner to extract printable ASCII string chunks.
### 2. Smart Classification (`classification/`)
* Combines statistical Machine Learning with keyword rule heuristics:
1. A **TF-IDF + Multinomial Logistic Regression** model trained on a localized dataset predicts the category.
2. If the prediction confidence falls below the **0.55 threshold**, the classifier invokes a fallback regular expression scanner searching for shipping terms (`dwt`, `laycan`, `dely`, `tct`, `loading port`).
* Categorizes target records into three strict groups:
* **Tonnage**: Open vessel availability details.
* **Cargo VC**: Voyage Charter cargo requirements.
* **Cargo TC**: Time Charter cargo requirements.
### 3. Case-Insensitive Data Extraction (`extraction/`)
Extracts custom fields depending on the assigned category:
* **Tonnage**: Vessel Name, Account Name, Open Port, Open Date, Vessel Type, Vessel Size, Flag, Built Year, Class, LOA, and Beam.
* **Cargo VC**: Cargo Name, Loading Port, Discharge Port, Laycan, Cargo Type, and Quantity.
* **Cargo TC**: Delivery Port, Redelivery Port, Duration, Laycan, Cargo Type, and Account.
* **Splitter Logic**: Automatically splits bulk emails containing multiple vessels or cargo listings (separated by `---` or `+++`) into individual database records.
### 4. Local Matching Engine (`routes/matching.py`)
Computes compatibility scores (scale `0–100`) between open vessels and charter requirements:
* **Port Proximity (Max 50 pts)**: Scored on exact string equivalence (50 pts), common token prefix matching (35 pts), partial containment (20 pts), or matching country codes (10 pts).
* **Laycan Overlap (Max 30 pts)**: Textual date parser extracts month/year combinations and grades month differences (0 month difference = 30 pts, 1 month = 20 pts).
* **Vessel Size vs Cargo Qty (Max 20 pts)**: Calculates the ratio between vessel DWT and cargo quantity (ratio between 0.6 and 1.6 = 20 pts).
### 5. SHA-256 Deduplication Layer (`routes/emails.py`)
* Strips all non-word symbols and spacing from the first 1000 characters of the body.
* Generates a unique SHA-256 hash representation.
* Compares incoming hashes against existing emails to mark and isolate broker forwards or circulars without halting processing.
### 6. Bidirectional Traceability & Verification (UX)
* **Email Split-Screen View**: The Inbox View presents a side-by-side verification interface showing the raw text next to the parsed records. Evaluators can confirm parser accuracy at a glance.
* **Source Trace-Backs**: Clicking the **βœ‰ (View Source Email)** icon next to any vessel or cargo record opens the original context modal instantly.
---
## Getting Started
To run the application smoothly on your local machine without high memory usage or compilation overhead, run the application directly through the pre-installed system Python.
### 1. Pre-population (Seeding)
Seed the database with sample real-world emails from Prime Maritime and other shipbrokers:
```bash
python3 seed.py
```
### 2. Running Automated Tests
Run the unit test suite covering classification, regex parser casings, matching weights, and deduplication:
```bash
python3 -m pytest tests/
```
### 3. Booting the Application
Start the Flask local development server:
```bash
PORT=8080 python3 app.py
```
Open **`http://127.0.0.1:8080`** in your web browser. *(Note: Do not open `index.html` as a file; always navigate through the served HTTP localhost address).*