Spaces:
Sleeping
Sleeping
|
Download README.md from archittmittal/Email-Segregation: direct link, hf CLI and curl.
- Browser
- Download file 6.7 kB
-
https://huggingface.co/spaces/archittmittal/Email-Segregation/resolve/main/README.md
- Command line
-
hf download hf://spaces/archittmittal/Email-Segregation/README.md
-
curl -L -o README.md https://huggingface.co/spaces/archittmittal/Email-Segregation/resolve/main/README.md
6.7 kB
| title: Email Segregation | |
| emoji: π§ | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: docker | |
| python_version: "3.9" | |
| app_file: app.py | |
| pinned: false | |
| # β ShipSeg: Automated Shipping Email Segregation & Data Extraction System | |
| [](https://email-segregation.vercel.app/) | |
| [](https://huggingface.co/spaces/archittmittal/Email-Segregation) | |
| ShipSeg is a modular, high-performance local pipeline built to ingest, classify, and extract commercial data from unstructured shipping broker emails and attachments. | |
| By automating the transition from messy emails to a structured database, the platform eliminates manual data entry, reduces vessel-to-cargo matching latency, and increases market visibility. | |
| --- | |
| ## π The Business Problem | |
| Maritime chartering desks and shipbrokers receive thousands of emails every day containing raw, unstandardized lists of **vessel availability (Tonnage)** and **cargo requirements (Voyage/Time Charters)**. Manually sorting these and typing them into a database leads to: | |
| * **Missed Opportunities**: High-value matches go unnoticed in overloaded inboxes. | |
| * **Operational Bottlenecks**: Significant latency in response time to broker offers. | |
| * **Data Inaccuracy**: Human error in capturing critical values (laycan dates, draft sizes, ports). | |
| --- | |
| ## β‘ System Constraints & Architecture | |
| > [!IMPORTANT] | |
| > **Zero Third-Party LLM Dependency Constraint**: To guarantee absolute data privacy, eliminate API subscription costs, and ensure instant offline latency, this system runs **entirely local machine learning models and heuristics**. It does **NOT** query OpenAI, Anthropic, or any external AI services. | |
| ### Technical Stack | |
| * **Backend**: Python 3.9+ / Flask (REST APIs & static routing) | |
| * **Local ML & NLP**: Scikit-Learn (TF-IDF Vectorization + Multinomial Logistic Regression) | |
| * **Data Layer**: SQLAlchemy ORM / SQLite | |
| * **Frontend**: Responsive Single Page Application (SPA) built using Vanilla JavaScript, HTML5, and a custom CSS premium dark-mode theme. | |
| --- | |
| ## π Pipeline Workflow | |
| ```mermaid | |
| flowchart TD | |
| A[Raw Incoming Email / Attachment] --> B[Ingestion Layer] | |
| B -->|Parse EML, TXT, or PDF| C[Deduplication Layer] | |
| C -->|SHA-256 Fingerprint Check| D{Is Duplicate?} | |
| D -->|Yes| E[Flag as Duplicate & Link Source] | |
| D -->|No| F[Classification Engine] | |
| F -->|Local ML / TF-IDF + Logistic Reg| G{Confidence > 55%?} | |
| G -->|No| H[Regex Rule Fallback Classifier] | |
| G -->|Yes| I[Category Route Matcher] | |
| H --> I | |
| I -->|Tonnage| J[Tonnage Extractor] | |
| I -->|Cargo VC| K[Cargo VC Extractor] | |
| I -->|Cargo TC| L[Cargo TC Extractor] | |
| J & K & L --> M[Database Persistence] | |
| M --> N[Matching Engine] | |
| N --> O[Ranked Vessel-Cargo Matching Score] | |
| ``` | |
| --- | |
| ## Core Module Specifications | |
| ### 1. Ingestion & Normalization (`ingestion/`) | |
| * **Multi-Format Support**: Parses raw email text paste blocks, standard `.eml` email files, `.txt` files, and `.pdf` attachments. | |
| * **Fallback PDF Parser**: Attempts extraction via `pdfminer.six`. If missing locally, it falls back to a custom binary regex stream scanner to extract printable ASCII string chunks. | |
| ### 2. Smart Classification (`classification/`) | |
| * Combines statistical Machine Learning with keyword rule heuristics: | |
| 1. A **TF-IDF + Multinomial Logistic Regression** model trained on a localized dataset predicts the category. | |
| 2. If the prediction confidence falls below the **0.55 threshold**, the classifier invokes a fallback regular expression scanner searching for shipping terms (`dwt`, `laycan`, `dely`, `tct`, `loading port`). | |
| * Categorizes target records into three strict groups: | |
| * **Tonnage**: Open vessel availability details. | |
| * **Cargo VC**: Voyage Charter cargo requirements. | |
| * **Cargo TC**: Time Charter cargo requirements. | |
| ### 3. Case-Insensitive Data Extraction (`extraction/`) | |
| Extracts custom fields depending on the assigned category: | |
| * **Tonnage**: Vessel Name, Account Name, Open Port, Open Date, Vessel Type, Vessel Size, Flag, Built Year, Class, LOA, and Beam. | |
| * **Cargo VC**: Cargo Name, Loading Port, Discharge Port, Laycan, Cargo Type, and Quantity. | |
| * **Cargo TC**: Delivery Port, Redelivery Port, Duration, Laycan, Cargo Type, and Account. | |
| * **Splitter Logic**: Automatically splits bulk emails containing multiple vessels or cargo listings (separated by `---` or `+++`) into individual database records. | |
| ### 4. Local Matching Engine (`routes/matching.py`) | |
| Computes compatibility scores (scale `0β100`) between open vessels and charter requirements: | |
| * **Port Proximity (Max 50 pts)**: Scored on exact string equivalence (50 pts), common token prefix matching (35 pts), partial containment (20 pts), or matching country codes (10 pts). | |
| * **Laycan Overlap (Max 30 pts)**: Textual date parser extracts month/year combinations and grades month differences (0 month difference = 30 pts, 1 month = 20 pts). | |
| * **Vessel Size vs Cargo Qty (Max 20 pts)**: Calculates the ratio between vessel DWT and cargo quantity (ratio between 0.6 and 1.6 = 20 pts). | |
| ### 5. SHA-256 Deduplication Layer (`routes/emails.py`) | |
| * Strips all non-word symbols and spacing from the first 1000 characters of the body. | |
| * Generates a unique SHA-256 hash representation. | |
| * Compares incoming hashes against existing emails to mark and isolate broker forwards or circulars without halting processing. | |
| ### 6. Bidirectional Traceability & Verification (UX) | |
| * **Email Split-Screen View**: The Inbox View presents a side-by-side verification interface showing the raw text next to the parsed records. Evaluators can confirm parser accuracy at a glance. | |
| * **Source Trace-Backs**: Clicking the **β (View Source Email)** icon next to any vessel or cargo record opens the original context modal instantly. | |
| --- | |
| ## Getting Started | |
| To run the application smoothly on your local machine without high memory usage or compilation overhead, run the application directly through the pre-installed system Python. | |
| ### 1. Pre-population (Seeding) | |
| Seed the database with sample real-world emails from Prime Maritime and other shipbrokers: | |
| ```bash | |
| python3 seed.py | |
| ``` | |
| ### 2. Running Automated Tests | |
| Run the unit test suite covering classification, regex parser casings, matching weights, and deduplication: | |
| ```bash | |
| python3 -m pytest tests/ | |
| ``` | |
| ### 3. Booting the Application | |
| Start the Flask local development server: | |
| ```bash | |
| PORT=8080 python3 app.py | |
| ``` | |
| Open **`http://127.0.0.1:8080`** in your web browser. *(Note: Do not open `index.html` as a file; always navigate through the served HTTP localhost address).* | |