Commit ·
81d06a3
1
Parent(s): 50ae2d7
docs: Complete Architecture Documentation (12 files) - 01-system-overview: High-level architecture diagram - 02-class-diagram: UML class diagram (Frontend + Backend) - 03-component-diagram: All modules and connections - 04-sequence-diagrams: 5 flows (Analysis, AutoComplete, Summarize, Save, Offline) - 05-dataflow-diagram: DFD Level 0 + Level 1 - 06-deployment-diagram: Docker + HF Spaces + CI/CD - 07-database-schema: ERD + RLS policies + triggers - 08-nlp-pipeline-diagram: Final NLP pipeline with all 5 models - 09-sync-engine-diagram: State machine + conflict resolution - 10-project-dependency-map: Full dependency graph - 11-production-architecture: Final production system diagram - 12-architecture-audit-report: Stats, risks, future expansion
Browse files- docs/architecture/01-system-overview.md +141 -0
- docs/architecture/02-class-diagram.md +312 -0
- docs/architecture/03-component-diagram.md +139 -0
- docs/architecture/04-sequence-diagrams.md +193 -0
- docs/architecture/05-dataflow-diagram.md +135 -0
- docs/architecture/06-deployment-diagram.md +130 -0
- docs/architecture/07-database-schema.md +148 -0
- docs/architecture/08-nlp-pipeline-diagram.md +198 -0
- docs/architecture/09-sync-engine-diagram.md +179 -0
- docs/architecture/10-project-dependency-map.md +195 -0
- docs/architecture/11-production-architecture.md +162 -0
- docs/architecture/12-architecture-audit-report.md +239 -0
docs/architecture/01-system-overview.md
ADDED
|
@@ -0,0 +1,141 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 01 — High-Level System Architecture
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
BAYAN is a full-stack Arabic NLP web application that provides intelligent writing assistance: spelling correction (AraSpell), grammar checking, punctuation restoration (PuncAra-v1), autocomplete suggestions, and text summarization. The system is deployed as a containerized Flask application on HuggingFace Spaces, with Supabase as the backend database and Google OAuth for authentication.
|
| 6 |
+
|
| 7 |
+
## Architecture Diagram
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
graph TD
|
| 11 |
+
subgraph "Client Layer"
|
| 12 |
+
U["👤 User (Browser)"]
|
| 13 |
+
UI["UI Layer<br/>Landing · Features · Editor · Pricing"]
|
| 14 |
+
EDITOR["Editor Engine<br/>editor.js · renderer.js · selection.js"]
|
| 15 |
+
AUTH["Auth Module<br/>Guest · Google OAuth · Supabase Auth"]
|
| 16 |
+
SYNC["Sync Engine<br/>SyncManager · SyncQueue · SyncResolver"]
|
| 17 |
+
DOCS["Documents Manager<br/>documents-api · documents-ui"]
|
| 18 |
+
SUMM_UI["Summaries UI<br/>summaries-api · summaries-ui"]
|
| 19 |
+
SETTINGS["Settings Sync<br/>settings-api · settings-sync"]
|
| 20 |
+
EXPORT["Export System<br/>TXT · DOCX · PDF"]
|
| 21 |
+
end
|
| 22 |
+
|
| 23 |
+
subgraph "API Layer (Flask)"
|
| 24 |
+
API["Flask REST API<br/>Gunicorn · CORS"]
|
| 25 |
+
HEALTH["/api/health"]
|
| 26 |
+
ANALYZE["/api/analyze"]
|
| 27 |
+
SPELL["/api/spelling"]
|
| 28 |
+
GRAMMAR["/api/grammar"]
|
| 29 |
+
PUNCT["/api/punctuation"]
|
| 30 |
+
SUMMARY["/api/summarize"]
|
| 31 |
+
AUTOCOMPLETE["/api/autocomplete"]
|
| 32 |
+
end
|
| 33 |
+
|
| 34 |
+
subgraph "NLP Layer (Models)"
|
| 35 |
+
ML["ModelLoader"]
|
| 36 |
+
ARASPELL["AraSpell<br/>AraBERT Encoder-Decoder"]
|
| 37 |
+
GRAM_MODEL["Bayan Grammar<br/>Grammar Rules Engine"]
|
| 38 |
+
PUNC_MODEL["PuncAra-v1<br/>Punctuation Restoration"]
|
| 39 |
+
AUTO_MODEL["AutoComplete<br/>Language Model"]
|
| 40 |
+
SUMM_MODEL["Summarization<br/>MBart Fine-tuned"]
|
| 41 |
+
end
|
| 42 |
+
|
| 43 |
+
subgraph "Data Layer"
|
| 44 |
+
SUPA["Supabase (PostgreSQL)"]
|
| 45 |
+
PROFILES["profiles"]
|
| 46 |
+
DOCUMENTS["documents"]
|
| 47 |
+
SUMMARIES["summaries"]
|
| 48 |
+
SETTINGS_DB["settings"]
|
| 49 |
+
LOCAL["localStorage<br/>Drafts · Dismissed Words"]
|
| 50 |
+
end
|
| 51 |
+
|
| 52 |
+
subgraph "Infrastructure"
|
| 53 |
+
HF["HuggingFace Spaces"]
|
| 54 |
+
DOCKER["Docker Container"]
|
| 55 |
+
GH["GitHub Repository"]
|
| 56 |
+
CI["GitHub Actions CI/CD"]
|
| 57 |
+
GOOGLE["Google OAuth Provider"]
|
| 58 |
+
end
|
| 59 |
+
|
| 60 |
+
U --> UI
|
| 61 |
+
UI --> EDITOR
|
| 62 |
+
UI --> AUTH
|
| 63 |
+
EDITOR --> SYNC
|
| 64 |
+
EDITOR --> DOCS
|
| 65 |
+
EDITOR --> SUMM_UI
|
| 66 |
+
EDITOR --> SETTINGS
|
| 67 |
+
EDITOR --> EXPORT
|
| 68 |
+
|
| 69 |
+
EDITOR -->|"POST /api/analyze"| API
|
| 70 |
+
SUMM_UI -->|"POST /api/summarize"| API
|
| 71 |
+
AUTH -->|"Supabase SDK"| SUPA
|
| 72 |
+
|
| 73 |
+
API --> HEALTH
|
| 74 |
+
API --> ANALYZE
|
| 75 |
+
API --> SPELL
|
| 76 |
+
API --> GRAMMAR
|
| 77 |
+
API --> PUNCT
|
| 78 |
+
API --> SUMMARY
|
| 79 |
+
API --> AUTOCOMPLETE
|
| 80 |
+
|
| 81 |
+
ANALYZE --> ML
|
| 82 |
+
SPELL --> ML
|
| 83 |
+
GRAMMAR --> ML
|
| 84 |
+
PUNCT --> ML
|
| 85 |
+
SUMMARY --> ML
|
| 86 |
+
AUTOCOMPLETE --> ML
|
| 87 |
+
|
| 88 |
+
ML --> ARASPELL
|
| 89 |
+
ML --> GRAM_MODEL
|
| 90 |
+
ML --> PUNC_MODEL
|
| 91 |
+
ML --> AUTO_MODEL
|
| 92 |
+
ML --> SUMM_MODEL
|
| 93 |
+
|
| 94 |
+
SYNC --> SUPA
|
| 95 |
+
DOCS --> SUPA
|
| 96 |
+
SUMM_UI --> SUPA
|
| 97 |
+
SETTINGS --> SUPA
|
| 98 |
+
|
| 99 |
+
SUPA --> PROFILES
|
| 100 |
+
SUPA --> DOCUMENTS
|
| 101 |
+
SUPA --> SUMMARIES
|
| 102 |
+
SUPA --> SETTINGS_DB
|
| 103 |
+
|
| 104 |
+
EDITOR --> LOCAL
|
| 105 |
+
|
| 106 |
+
HF --> DOCKER
|
| 107 |
+
DOCKER --> API
|
| 108 |
+
GH --> CI
|
| 109 |
+
CI -->|"Deploy"| HF
|
| 110 |
+
AUTH --> GOOGLE
|
| 111 |
+
|
| 112 |
+
style U fill:#4F46E5,stroke:#312E81,color:#fff
|
| 113 |
+
style API fill:#059669,stroke:#064E3B,color:#fff
|
| 114 |
+
style SUPA fill:#3B82F6,stroke:#1E3A8A,color:#fff
|
| 115 |
+
style HF fill:#FF9D00,stroke:#92400E,color:#fff
|
| 116 |
+
style ML fill:#7C3AED,stroke:#4C1D95,color:#fff
|
| 117 |
+
```
|
| 118 |
+
|
| 119 |
+
## Layer Descriptions
|
| 120 |
+
|
| 121 |
+
| Layer | Components | Responsibility |
|
| 122 |
+
|-------|-----------|----------------|
|
| 123 |
+
| **Client** | HTML/CSS/JS SPA | User interface, editor, auth gate, document management |
|
| 124 |
+
| **API** | Flask + Gunicorn | REST endpoints, request validation, model orchestration |
|
| 125 |
+
| **NLP** | 5 ML models | Spelling, grammar, punctuation, autocomplete, summarization |
|
| 126 |
+
| **Data** | Supabase + localStorage | Persistent storage, auth, offline drafts |
|
| 127 |
+
| **Infrastructure** | Docker + HF Spaces + CI | Containerization, deployment, monitoring |
|
| 128 |
+
|
| 129 |
+
## Design Rationale
|
| 130 |
+
|
| 131 |
+
1. **Single-Page Application (SPA)**: No framework overhead — pure HTML/CSS/JS for minimal bundle size and fast load.
|
| 132 |
+
2. **Flask API**: Lightweight Python backend, ideal for ML model serving.
|
| 133 |
+
3. **Supabase**: Managed PostgreSQL with built-in auth, RLS, and realtime — eliminates custom backend for CRUD.
|
| 134 |
+
4. **Docker**: Ensures model dependencies are pre-cached at build time (no runtime downloads).
|
| 135 |
+
5. **HuggingFace Spaces**: Free GPU/CPU hosting optimized for ML model serving.
|
| 136 |
+
|
| 137 |
+
## Extension Points
|
| 138 |
+
|
| 139 |
+
- Additional NLP models can be added via `ModelLoader` without changing the API structure.
|
| 140 |
+
- New Supabase tables follow the same RLS pattern.
|
| 141 |
+
- Frontend pages follow a `showPage()` pattern for easy addition.
|
docs/architecture/02-class-diagram.md
ADDED
|
@@ -0,0 +1,312 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 02 — UML Class Diagram
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
This diagram models the complete class structure of the BAYAN system across Frontend and Backend layers, showing composition, aggregation, inheritance, and dependency relationships.
|
| 6 |
+
|
| 7 |
+
## Class Diagram
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
classDiagram
|
| 11 |
+
direction TB
|
| 12 |
+
|
| 13 |
+
%% ─── Frontend: Editor Core ───
|
| 14 |
+
class Editor {
|
| 15 |
+
-analyzeTimeout: Timer
|
| 16 |
+
-analyzeAbortController: AbortController
|
| 17 |
+
-undoStack: Array
|
| 18 |
+
-redoStack: Array
|
| 19 |
+
-dismissedWords: Set
|
| 20 |
+
+initEditor()
|
| 21 |
+
+getEditorText(): string
|
| 22 |
+
+setEditorHTML(html)
|
| 23 |
+
+updateEditorStats()
|
| 24 |
+
+analyzeTextDelayed()
|
| 25 |
+
+applyCorrection()
|
| 26 |
+
+dismissSuggestion(suggestion)
|
| 27 |
+
+applyAllSuggestions()
|
| 28 |
+
+pushUndoState()
|
| 29 |
+
+editorUndo(): boolean
|
| 30 |
+
+editorRedo(): boolean
|
| 31 |
+
+clearEditor()
|
| 32 |
+
+copyText()
|
| 33 |
+
}
|
| 34 |
+
|
| 35 |
+
class Renderer {
|
| 36 |
+
+renderCorrections(text, suggestions)
|
| 37 |
+
+highlightErrors(editor, suggestions)
|
| 38 |
+
+createErrorSpan(text, type, id)
|
| 39 |
+
+escapeHtml(text): string
|
| 40 |
+
+renderSuggestionsList(suggestions)
|
| 41 |
+
+renderSuggestionCard(suggestion, idx)
|
| 42 |
+
}
|
| 43 |
+
|
| 44 |
+
class SelectionManager {
|
| 45 |
+
+saveSelection(): Range
|
| 46 |
+
+restoreSelection(range)
|
| 47 |
+
+getCaretOffset(): number
|
| 48 |
+
+setCaretOffset(offset)
|
| 49 |
+
+getSelectedText(): string
|
| 50 |
+
}
|
| 51 |
+
|
| 52 |
+
class FormatManager {
|
| 53 |
+
+execFormat(command, value)
|
| 54 |
+
+toggleBold()
|
| 55 |
+
+toggleItalic()
|
| 56 |
+
+toggleUnderline()
|
| 57 |
+
+setFontFamily(font)
|
| 58 |
+
+setFontSize(size)
|
| 59 |
+
+setTextColor(color)
|
| 60 |
+
+setHighlightColor(color)
|
| 61 |
+
+setAlignment(align)
|
| 62 |
+
+insertList(type)
|
| 63 |
+
+clearFormatting()
|
| 64 |
+
+initDocSearch()
|
| 65 |
+
}
|
| 66 |
+
|
| 67 |
+
%% ─── Frontend: Auth ───
|
| 68 |
+
class AuthManager {
|
| 69 |
+
+signInAsGuest(): Promise
|
| 70 |
+
+signInWithGoogle(): Promise
|
| 71 |
+
+linkGoogle(): Promise
|
| 72 |
+
+signOut(): Promise
|
| 73 |
+
+getDisplayName(user): string
|
| 74 |
+
+getAuthProvider(user): string
|
| 75 |
+
+getAvatarUrl(user): string
|
| 76 |
+
+isGuestUser(user): boolean
|
| 77 |
+
}
|
| 78 |
+
|
| 79 |
+
class AuthUI {
|
| 80 |
+
+bindAuthUIEvents()
|
| 81 |
+
+showAuthGate()
|
| 82 |
+
+hideAuthGate()
|
| 83 |
+
+updateAuthUI(user)
|
| 84 |
+
+closeAuthMenu()
|
| 85 |
+
+showAuthOfflineBanner(show)
|
| 86 |
+
}
|
| 87 |
+
|
| 88 |
+
class SessionManager {
|
| 89 |
+
+initSession()
|
| 90 |
+
+onAuthStateChange(callback)
|
| 91 |
+
+getSession(): Session
|
| 92 |
+
+refreshSession()
|
| 93 |
+
}
|
| 94 |
+
|
| 95 |
+
class SupabaseClient {
|
| 96 |
+
+getSupabaseClient(): Client
|
| 97 |
+
-url: string
|
| 98 |
+
-anonKey: string
|
| 99 |
+
}
|
| 100 |
+
|
| 101 |
+
%% ─── Frontend: Documents ───
|
| 102 |
+
class DocumentManager {
|
| 103 |
+
+createDocument(title, content): Promise
|
| 104 |
+
+loadDocuments(): Promise
|
| 105 |
+
+loadDocument(id): Promise
|
| 106 |
+
+saveDocument(id, content): Promise
|
| 107 |
+
+renameDocument(id, title): Promise
|
| 108 |
+
+deleteDocument(id): Promise
|
| 109 |
+
}
|
| 110 |
+
|
| 111 |
+
class DocumentUI {
|
| 112 |
+
-docState: DocState
|
| 113 |
+
+setDocState(state)
|
| 114 |
+
+getDocState(): DocState
|
| 115 |
+
+renderDocList(docs)
|
| 116 |
+
+openDocument(id)
|
| 117 |
+
+updateTitleBar()
|
| 118 |
+
+loadAndRenderList()
|
| 119 |
+
}
|
| 120 |
+
|
| 121 |
+
class DocState {
|
| 122 |
+
+currentDocumentId: string
|
| 123 |
+
+currentDocumentTitle: string
|
| 124 |
+
+hasUnsavedChanges: boolean
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
%% ─── Frontend: Summaries ───
|
| 128 |
+
class SummaryManager {
|
| 129 |
+
+createSummary(docId, originalText, summaryText): Promise
|
| 130 |
+
+loadSummaries(docId): Promise
|
| 131 |
+
+deleteSummary(id): Promise
|
| 132 |
+
}
|
| 133 |
+
|
| 134 |
+
class SummaryUI {
|
| 135 |
+
+renderSummaryList(summaries)
|
| 136 |
+
+showSummaryPanel(summary)
|
| 137 |
+
+exportSummaryAsTxt()
|
| 138 |
+
}
|
| 139 |
+
|
| 140 |
+
%% ─── Frontend: Settings ───
|
| 141 |
+
class SettingsAPI {
|
| 142 |
+
+loadSettings(): Promise
|
| 143 |
+
+saveSettings(settings): Promise
|
| 144 |
+
}
|
| 145 |
+
|
| 146 |
+
class SettingsSync {
|
| 147 |
+
+initSettingsSync()
|
| 148 |
+
+applySettings(settings)
|
| 149 |
+
+getLocalSettings(): object
|
| 150 |
+
}
|
| 151 |
+
|
| 152 |
+
%% ─── Frontend: Sync ───
|
| 153 |
+
class SyncManager {
|
| 154 |
+
-syncInterval: Timer
|
| 155 |
+
-isOnline: boolean
|
| 156 |
+
+initSync()
|
| 157 |
+
+scheduleSave()
|
| 158 |
+
+syncNow(): Promise
|
| 159 |
+
+handleOnline()
|
| 160 |
+
+handleOffline()
|
| 161 |
+
}
|
| 162 |
+
|
| 163 |
+
class SyncQueue {
|
| 164 |
+
-queue: Array
|
| 165 |
+
+enqueue(operation)
|
| 166 |
+
+dequeue(): Operation
|
| 167 |
+
+peek(): Operation
|
| 168 |
+
+isEmpty(): boolean
|
| 169 |
+
+flush(): Promise
|
| 170 |
+
}
|
| 171 |
+
|
| 172 |
+
class SyncResolver {
|
| 173 |
+
+resolve(local, remote): Resolved
|
| 174 |
+
+mergeConflict(a, b): string
|
| 175 |
+
+lastWriteWins(a, b): object
|
| 176 |
+
}
|
| 177 |
+
|
| 178 |
+
%% ─── Frontend: UI ───
|
| 179 |
+
class UIManager {
|
| 180 |
+
+showPage(pageId)
|
| 181 |
+
+switchTab(tabId)
|
| 182 |
+
+showToast(message, type)
|
| 183 |
+
+setAnalyzingState(isAnalyzing)
|
| 184 |
+
+updateSuggestionCounts(s, g, p)
|
| 185 |
+
+updateWritingScore(s, g, p)
|
| 186 |
+
+renderSuggestionsList(suggestions)
|
| 187 |
+
}
|
| 188 |
+
|
| 189 |
+
class ThemeManager {
|
| 190 |
+
+initTheme()
|
| 191 |
+
+toggleTheme()
|
| 192 |
+
+getPreferredTheme(): string
|
| 193 |
+
+applyTheme(theme)
|
| 194 |
+
}
|
| 195 |
+
|
| 196 |
+
class APIClient {
|
| 197 |
+
+apiPost(endpoint, body): Promise
|
| 198 |
+
+apiGet(endpoint): Promise
|
| 199 |
+
-baseUrl: string
|
| 200 |
+
}
|
| 201 |
+
|
| 202 |
+
%% ─── Backend: Flask ───
|
| 203 |
+
class FlaskApp {
|
| 204 |
+
+index(): Response
|
| 205 |
+
+health_check(): JSON
|
| 206 |
+
+debug_models(): JSON
|
| 207 |
+
+analyze_text(): JSON
|
| 208 |
+
+spelling_correction(): JSON
|
| 209 |
+
+grammar_correction(): JSON
|
| 210 |
+
+add_punctuation(): JSON
|
| 211 |
+
+summarize(): JSON
|
| 212 |
+
+autocomplete(): JSON
|
| 213 |
+
}
|
| 214 |
+
|
| 215 |
+
class ModelLoader {
|
| 216 |
+
+load_models()
|
| 217 |
+
+ensure_models_loaded()
|
| 218 |
+
-summarization_model: SummarizationModel
|
| 219 |
+
-spelling_model: SpellingModel
|
| 220 |
+
-grammar_model: GrammarModel
|
| 221 |
+
-punctuation_model: PunctuationModel
|
| 222 |
+
-autocomplete_model: AutocompleteModel
|
| 223 |
+
}
|
| 224 |
+
|
| 225 |
+
%% ─── Backend: NLP Services ───
|
| 226 |
+
class AraSpellService {
|
| 227 |
+
+correct(text): CorrectionResult
|
| 228 |
+
+get_alternatives(word): List
|
| 229 |
+
-model: EncoderDecoderModel
|
| 230 |
+
-tokenizer: AutoTokenizer
|
| 231 |
+
}
|
| 232 |
+
|
| 233 |
+
class GrammarService {
|
| 234 |
+
+check(text): List~GrammarError~
|
| 235 |
+
+suggest_fix(error): string
|
| 236 |
+
-rules: GrammarRules
|
| 237 |
+
}
|
| 238 |
+
|
| 239 |
+
class PunctuationService {
|
| 240 |
+
+restore(text): string
|
| 241 |
+
-model: PunctuationModel
|
| 242 |
+
}
|
| 243 |
+
|
| 244 |
+
class AutoCompleteService {
|
| 245 |
+
+complete(prefix): List~Suggestion~
|
| 246 |
+
-model: LanguageModel
|
| 247 |
+
}
|
| 248 |
+
|
| 249 |
+
class SummarizationService {
|
| 250 |
+
+summarize(text, max_length): string
|
| 251 |
+
-model: MBartForConditionalGeneration
|
| 252 |
+
-tokenizer: AutoTokenizer
|
| 253 |
+
}
|
| 254 |
+
|
| 255 |
+
class HFInference {
|
| 256 |
+
+hf_summarize(text): string
|
| 257 |
+
+hf_correct_spelling(text): string
|
| 258 |
+
+hf_add_punctuation(text): string
|
| 259 |
+
+hf_autocomplete(text): string
|
| 260 |
+
+check_hf_api_available(): boolean
|
| 261 |
+
}
|
| 262 |
+
|
| 263 |
+
%% ─── Relationships ───
|
| 264 |
+
Editor *-- SelectionManager : uses
|
| 265 |
+
Editor *-- Renderer : renders via
|
| 266 |
+
Editor o-- FormatManager : formatting
|
| 267 |
+
Editor --> APIClient : sends text
|
| 268 |
+
Editor --> SyncManager : triggers save
|
| 269 |
+
|
| 270 |
+
AuthManager --> SupabaseClient : authenticates via
|
| 271 |
+
AuthManager --> SessionManager : manages session
|
| 272 |
+
AuthUI --> AuthManager : calls
|
| 273 |
+
|
| 274 |
+
DocumentManager --> SupabaseClient : CRUD
|
| 275 |
+
DocumentUI --> DocumentManager : delegates to
|
| 276 |
+
DocumentUI *-- DocState : owns
|
| 277 |
+
|
| 278 |
+
SummaryManager --> SupabaseClient : CRUD
|
| 279 |
+
SummaryUI --> SummaryManager : delegates to
|
| 280 |
+
|
| 281 |
+
SettingsSync --> SettingsAPI : persists via
|
| 282 |
+
SettingsAPI --> SupabaseClient : CRUD
|
| 283 |
+
|
| 284 |
+
SyncManager *-- SyncQueue : owns
|
| 285 |
+
SyncManager *-- SyncResolver : resolves with
|
| 286 |
+
SyncManager --> DocumentManager : saves via
|
| 287 |
+
|
| 288 |
+
UIManager --> Editor : controls
|
| 289 |
+
UIManager --> Renderer : renders
|
| 290 |
+
|
| 291 |
+
FlaskApp *-- ModelLoader : loads models
|
| 292 |
+
FlaskApp --> AraSpellService : spelling
|
| 293 |
+
FlaskApp --> GrammarService : grammar
|
| 294 |
+
FlaskApp --> PunctuationService : punctuation
|
| 295 |
+
FlaskApp --> AutoCompleteService : autocomplete
|
| 296 |
+
FlaskApp --> SummarizationService : summarization
|
| 297 |
+
FlaskApp --> HFInference : remote fallback
|
| 298 |
+
|
| 299 |
+
ModelLoader --> AraSpellService : initializes
|
| 300 |
+
ModelLoader --> GrammarService : initializes
|
| 301 |
+
ModelLoader --> PunctuationService : initializes
|
| 302 |
+
ModelLoader --> AutoCompleteService : initializes
|
| 303 |
+
ModelLoader --> SummarizationService : initializes
|
| 304 |
+
```
|
| 305 |
+
|
| 306 |
+
## Design Rationale
|
| 307 |
+
|
| 308 |
+
- **Composition (`*--`)**: Editor owns SelectionManager and Renderer — they cannot exist without the Editor.
|
| 309 |
+
- **Aggregation (`o--`)**: FormatManager is reusable and independent.
|
| 310 |
+
- **Dependency (`-->`)**: API calls and data flows between modules.
|
| 311 |
+
- **Frontend modules** are globally-scoped functions (vanilla JS) but modeled as classes for UML clarity.
|
| 312 |
+
- **Backend** uses Flask route functions modeled as FlaskApp methods, with NLP services as separate classes.
|
docs/architecture/03-component-diagram.md
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 03 — Component Diagram
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
This diagram shows how BAYAN's software components are organized and interact across Frontend, API, NLP, and Data layers.
|
| 6 |
+
|
| 7 |
+
## Component Diagram
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
graph TB
|
| 11 |
+
subgraph "Frontend SPA"
|
| 12 |
+
direction TB
|
| 13 |
+
subgraph "Pages"
|
| 14 |
+
HOME["🏠 Landing Page"]
|
| 15 |
+
FEATURES["⚡ Features Page"]
|
| 16 |
+
EDITOR_PAGE["📝 Editor Page"]
|
| 17 |
+
PRICING["💰 Pricing Page"]
|
| 18 |
+
P404["🚫 404 Page"]
|
| 19 |
+
end
|
| 20 |
+
|
| 21 |
+
subgraph "Editor Core"
|
| 22 |
+
EDITOR["editor.js<br/>State · Undo/Redo · Analysis"]
|
| 23 |
+
RENDERER["renderer.js<br/>Error Highlighting · Spans"]
|
| 24 |
+
SELECTION["selection.js<br/>Caret · Range · Offset"]
|
| 25 |
+
FORMAT["format.js<br/>Bold · Italic · Lists · Colors"]
|
| 26 |
+
end
|
| 27 |
+
|
| 28 |
+
subgraph "Auth System"
|
| 29 |
+
AUTH_GATE["Auth Gate<br/>Login Modal"]
|
| 30 |
+
AUTH_JS["auth.js<br/>Guest · Google · Link"]
|
| 31 |
+
AUTH_UI["auth-ui.js<br/>Menu · Avatar · Banner"]
|
| 32 |
+
SESSION["session.js<br/>Token Refresh"]
|
| 33 |
+
SB_CLIENT["client.js<br/>Supabase Init"]
|
| 34 |
+
end
|
| 35 |
+
|
| 36 |
+
subgraph "Data Modules"
|
| 37 |
+
DOC_API["documents-api.js<br/>CRUD Operations"]
|
| 38 |
+
DOC_UI["documents-ui.js<br/>Sidebar · List · Search"]
|
| 39 |
+
DOC_STATE["documents-state.js<br/>Current Doc Tracking"]
|
| 40 |
+
SUMM_API["summaries-api.js<br/>Save · Load · Delete"]
|
| 41 |
+
SUMM_UI_MOD["summaries-ui.js<br/>Display · Export TXT"]
|
| 42 |
+
SET_API["settings-api.js<br/>Load · Save"]
|
| 43 |
+
SET_SYNC["settings-sync.js<br/>Apply · Sync"]
|
| 44 |
+
end
|
| 45 |
+
|
| 46 |
+
subgraph "Sync Engine"
|
| 47 |
+
SYNC_MGR["sync-manager.js<br/>Online/Offline"]
|
| 48 |
+
SYNC_Q["sync-queue.js<br/>Operation Queue"]
|
| 49 |
+
SYNC_RES["sync-resolver.js<br/>Conflict Resolution"]
|
| 50 |
+
end
|
| 51 |
+
|
| 52 |
+
subgraph "Utilities"
|
| 53 |
+
API_JS["api.js<br/>HTTP Client"]
|
| 54 |
+
THEME["theme.js<br/>Dark/Light Toggle"]
|
| 55 |
+
UI_JS["ui.js<br/>Toast · Score · Suggestions"]
|
| 56 |
+
end
|
| 57 |
+
end
|
| 58 |
+
|
| 59 |
+
subgraph "API Layer (Flask)"
|
| 60 |
+
FLASK["Flask App<br/>app.py"]
|
| 61 |
+
|
| 62 |
+
subgraph "Endpoints"
|
| 63 |
+
E_HEALTH["/api/health"]
|
| 64 |
+
E_ANALYZE["/api/analyze"]
|
| 65 |
+
E_SPELL["/api/spelling"]
|
| 66 |
+
E_GRAMMAR["/api/grammar"]
|
| 67 |
+
E_PUNCT["/api/punctuation"]
|
| 68 |
+
E_SUMMARY["/api/summarize"]
|
| 69 |
+
E_AUTO["/api/autocomplete"]
|
| 70 |
+
E_DEBUG["/api/debug/models"]
|
| 71 |
+
end
|
| 72 |
+
end
|
| 73 |
+
|
| 74 |
+
subgraph "NLP Services"
|
| 75 |
+
LOADER["model_loader.py<br/>Lazy Loading · Caching"]
|
| 76 |
+
|
| 77 |
+
subgraph "Models"
|
| 78 |
+
M_SPELL["AraSpell<br/>araspell_service.py<br/>araspell_rules.py"]
|
| 79 |
+
M_GRAM["Grammar Service<br/>grammar_service.py"]
|
| 80 |
+
M_PUNCT["PuncAra-v1<br/>Punctuation Model"]
|
| 81 |
+
M_AUTO["AutoComplete<br/>Language Model"]
|
| 82 |
+
M_SUMM["Summarization<br/>MBart (float16)"]
|
| 83 |
+
end
|
| 84 |
+
|
| 85 |
+
HF_API["hf_inference.py<br/>Remote HF API Fallback"]
|
| 86 |
+
end
|
| 87 |
+
|
| 88 |
+
subgraph "Data Layer"
|
| 89 |
+
SUPABASE["Supabase<br/>PostgreSQL + Auth + RLS"]
|
| 90 |
+
LOCALSTORAGE["localStorage<br/>Drafts · Goals · Dismissed"]
|
| 91 |
+
end
|
| 92 |
+
|
| 93 |
+
%% Connections
|
| 94 |
+
EDITOR --> API_JS
|
| 95 |
+
API_JS --> FLASK
|
| 96 |
+
FLASK --> E_HEALTH & E_ANALYZE & E_SPELL & E_GRAMMAR & E_PUNCT & E_SUMMARY & E_AUTO
|
| 97 |
+
|
| 98 |
+
E_ANALYZE --> LOADER
|
| 99 |
+
E_SPELL --> LOADER
|
| 100 |
+
E_GRAMMAR --> LOADER
|
| 101 |
+
E_PUNCT --> LOADER
|
| 102 |
+
E_SUMMARY --> LOADER
|
| 103 |
+
E_AUTO --> LOADER
|
| 104 |
+
|
| 105 |
+
LOADER --> M_SPELL & M_GRAM & M_PUNCT & M_AUTO & M_SUMM
|
| 106 |
+
LOADER --> HF_API
|
| 107 |
+
|
| 108 |
+
DOC_API --> SUPABASE
|
| 109 |
+
SUMM_API --> SUPABASE
|
| 110 |
+
SET_API --> SUPABASE
|
| 111 |
+
AUTH_JS --> SUPABASE
|
| 112 |
+
SYNC_MGR --> DOC_API
|
| 113 |
+
|
| 114 |
+
EDITOR --> LOCALSTORAGE
|
| 115 |
+
|
| 116 |
+
style FLASK fill:#059669,color:#fff
|
| 117 |
+
style SUPABASE fill:#3B82F6,color:#fff
|
| 118 |
+
style LOADER fill:#7C3AED,color:#fff
|
| 119 |
+
```
|
| 120 |
+
|
| 121 |
+
## Component Responsibilities
|
| 122 |
+
|
| 123 |
+
| Component | Files | Responsibility |
|
| 124 |
+
|-----------|-------|----------------|
|
| 125 |
+
| Editor Core | `editor.js`, `renderer.js`, `selection.js`, `format.js` | Text editing, error highlighting, formatting |
|
| 126 |
+
| Auth System | `auth.js`, `auth-ui.js`, `session.js`, `client.js`, `config.js` | Authentication flow, session management |
|
| 127 |
+
| Documents | `documents-api.js`, `documents-ui.js`, `documents-state.js` | Document CRUD, sidebar, search |
|
| 128 |
+
| Summaries | `summaries-api.js`, `summaries-ui.js` | Summary storage and display |
|
| 129 |
+
| Settings | `settings-api.js`, `settings-sync.js` | User preferences persistence |
|
| 130 |
+
| Sync Engine | `sync-manager.js`, `sync-queue.js`, `sync-resolver.js` | Offline support, conflict resolution |
|
| 131 |
+
| API Client | `api.js` | HTTP request abstraction |
|
| 132 |
+
| Flask API | `app.py` | 8 REST endpoints |
|
| 133 |
+
| NLP Services | `model_loader.py`, `araspell_service.py`, `hf_inference.py` | Model loading, inference |
|
| 134 |
+
|
| 135 |
+
## Extension Points
|
| 136 |
+
|
| 137 |
+
- New pages: Add HTML section + `showPage()` entry.
|
| 138 |
+
- New NLP model: Add to `model_loader.py` + new `/api/` route.
|
| 139 |
+
- New Supabase table: Add migration SQL + API module + UI module.
|
docs/architecture/04-sequence-diagrams.md
ADDED
|
@@ -0,0 +1,193 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 04 — Sequence Diagrams
|
| 2 |
+
|
| 3 |
+
## A. Typing + NLP Analysis Flow
|
| 4 |
+
|
| 5 |
+
```mermaid
|
| 6 |
+
sequenceDiagram
|
| 7 |
+
participant U as User
|
| 8 |
+
participant E as Editor (editor.js)
|
| 9 |
+
participant API as api.js
|
| 10 |
+
participant F as Flask (/api/analyze)
|
| 11 |
+
participant SP as AraSpell
|
| 12 |
+
participant GR as Grammar
|
| 13 |
+
participant PU as Punctuation
|
| 14 |
+
participant R as Renderer
|
| 15 |
+
|
| 16 |
+
U->>E: Types text
|
| 17 |
+
E->>E: Debounce (1000ms)
|
| 18 |
+
E->>E: updateEditorStats()
|
| 19 |
+
E->>E: Save draft to localStorage
|
| 20 |
+
|
| 21 |
+
E->>API: POST /api/analyze {text}
|
| 22 |
+
API->>F: HTTP POST
|
| 23 |
+
|
| 24 |
+
F->>F: Validate text length (≤5000 chars)
|
| 25 |
+
F->>SP: correct(text)
|
| 26 |
+
SP-->>F: {corrections[], corrected_text}
|
| 27 |
+
|
| 28 |
+
F->>GR: check(corrected_text)
|
| 29 |
+
GR-->>F: {grammar_errors[]}
|
| 30 |
+
|
| 31 |
+
F->>PU: restore(corrected_text)
|
| 32 |
+
PU-->>F: {punctuated_text, punctuation_changes[]}
|
| 33 |
+
|
| 34 |
+
F->>F: Merge all suggestions
|
| 35 |
+
F->>F: Map offsets back to original text
|
| 36 |
+
F-->>API: {suggestions[], score}
|
| 37 |
+
API-->>E: JSON Response
|
| 38 |
+
|
| 39 |
+
E->>R: highlightErrors(editor, suggestions)
|
| 40 |
+
R->>R: Create <span> elements with error classes
|
| 41 |
+
E->>E: updateSuggestionCounts()
|
| 42 |
+
E->>E: updateWritingScore()
|
| 43 |
+
R->>R: renderSuggestionsList() in sidebar
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
---
|
| 47 |
+
|
| 48 |
+
## B. AutoComplete Flow
|
| 49 |
+
|
| 50 |
+
```mermaid
|
| 51 |
+
sequenceDiagram
|
| 52 |
+
participant U as User
|
| 53 |
+
participant E as Editor
|
| 54 |
+
participant API as api.js
|
| 55 |
+
participant F as Flask (/api/autocomplete)
|
| 56 |
+
participant AC as AutoComplete Model
|
| 57 |
+
|
| 58 |
+
U->>E: Types partial word
|
| 59 |
+
E->>E: Detect word boundary
|
| 60 |
+
E->>API: POST /api/autocomplete {text, cursor_position}
|
| 61 |
+
API->>F: HTTP POST
|
| 62 |
+
|
| 63 |
+
F->>AC: complete(prefix_text)
|
| 64 |
+
AC->>AC: Generate top-k suggestions
|
| 65 |
+
AC-->>F: {suggestions: ["word1", "word2", "word3"]}
|
| 66 |
+
|
| 67 |
+
F-->>API: JSON Response
|
| 68 |
+
API-->>E: Suggestions array
|
| 69 |
+
|
| 70 |
+
E->>E: Show autocomplete dropdown
|
| 71 |
+
U->>E: Select suggestion (click/Tab)
|
| 72 |
+
E->>E: Insert selected text at cursor
|
| 73 |
+
E->>E: Close dropdown
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
---
|
| 77 |
+
|
| 78 |
+
## C. Summarization Flow
|
| 79 |
+
|
| 80 |
+
```mermaid
|
| 81 |
+
sequenceDiagram
|
| 82 |
+
participant U as User
|
| 83 |
+
participant UI as SummaryUI
|
| 84 |
+
participant API as api.js
|
| 85 |
+
participant F as Flask (/api/summarize)
|
| 86 |
+
participant SM as Summarization Model (MBart)
|
| 87 |
+
participant SA as summaries-api.js
|
| 88 |
+
participant SB as Supabase
|
| 89 |
+
|
| 90 |
+
U->>UI: Click "تلخيص" tab
|
| 91 |
+
U->>UI: Click "لخّص النص"
|
| 92 |
+
UI->>UI: Get editor text
|
| 93 |
+
|
| 94 |
+
UI->>API: POST /api/summarize {text, max_length}
|
| 95 |
+
API->>F: HTTP POST
|
| 96 |
+
|
| 97 |
+
F->>F: Validate text length (≥10 chars)
|
| 98 |
+
F->>SM: summarize(text, max_length=512)
|
| 99 |
+
SM->>SM: Tokenize (ar_AR)
|
| 100 |
+
SM->>SM: Generate summary (beam search)
|
| 101 |
+
SM-->>F: {summary_text}
|
| 102 |
+
|
| 103 |
+
F-->>API: JSON {summary, word_count, compression_ratio}
|
| 104 |
+
API-->>UI: Response
|
| 105 |
+
|
| 106 |
+
UI->>UI: Display summary in panel
|
| 107 |
+
UI->>SA: createSummary(docId, original, summary)
|
| 108 |
+
SA->>SB: INSERT INTO summaries
|
| 109 |
+
SB-->>SA: {id, created_at}
|
| 110 |
+
SA-->>UI: Success
|
| 111 |
+
|
| 112 |
+
U->>UI: Click "تصدير TXT"
|
| 113 |
+
UI->>UI: Download summary as .txt file
|
| 114 |
+
```
|
| 115 |
+
|
| 116 |
+
---
|
| 117 |
+
|
| 118 |
+
## D. Document Save Flow
|
| 119 |
+
|
| 120 |
+
```mermaid
|
| 121 |
+
sequenceDiagram
|
| 122 |
+
participant U as User
|
| 123 |
+
participant E as Editor
|
| 124 |
+
participant SM as SyncManager
|
| 125 |
+
participant SQ as SyncQueue
|
| 126 |
+
participant DA as documents-api.js
|
| 127 |
+
participant SB as Supabase
|
| 128 |
+
|
| 129 |
+
U->>E: Types / Edits text
|
| 130 |
+
E->>E: Save to localStorage (draft)
|
| 131 |
+
E->>SM: scheduleSave()
|
| 132 |
+
SM->>SM: Debounce (2000ms)
|
| 133 |
+
|
| 134 |
+
SM->>SM: Check isAuthenticated
|
| 135 |
+
alt Authenticated
|
| 136 |
+
SM->>DA: saveDocument(docId, content)
|
| 137 |
+
DA->>SB: UPDATE documents SET content WHERE id
|
| 138 |
+
SB-->>DA: Success
|
| 139 |
+
DA-->>SM: true
|
| 140 |
+
SM->>SM: showAutoSaveStatus("تم الحفظ")
|
| 141 |
+
else Offline
|
| 142 |
+
SM->>SQ: enqueue({type: "save", docId, content})
|
| 143 |
+
SQ->>SQ: Store in memory
|
| 144 |
+
SM->>SM: showAutoSaveStatus("سيتم الحفظ عند الاتصال")
|
| 145 |
+
end
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
---
|
| 149 |
+
|
| 150 |
+
## E. Offline Recovery Flow
|
| 151 |
+
|
| 152 |
+
```mermaid
|
| 153 |
+
sequenceDiagram
|
| 154 |
+
participant SM as SyncManager
|
| 155 |
+
participant SQ as SyncQueue
|
| 156 |
+
participant SR as SyncResolver
|
| 157 |
+
participant DA as documents-api.js
|
| 158 |
+
participant SB as Supabase
|
| 159 |
+
|
| 160 |
+
Note over SM: User comes back online
|
| 161 |
+
|
| 162 |
+
SM->>SM: handleOnline() triggered
|
| 163 |
+
SM->>SQ: isEmpty()?
|
| 164 |
+
|
| 165 |
+
alt Queue has pending operations
|
| 166 |
+
loop For each queued operation
|
| 167 |
+
SQ->>SM: dequeue() → operation
|
| 168 |
+
SM->>DA: loadDocument(op.docId)
|
| 169 |
+
DA->>SB: SELECT content FROM documents WHERE id
|
| 170 |
+
SB-->>DA: {remote_content, updated_at}
|
| 171 |
+
|
| 172 |
+
SM->>SR: resolve(local_content, remote_content)
|
| 173 |
+
alt No conflict
|
| 174 |
+
SR-->>SM: Use local version
|
| 175 |
+
else Conflict detected
|
| 176 |
+
SR->>SR: lastWriteWins(local, remote)
|
| 177 |
+
SR-->>SM: Resolved content
|
| 178 |
+
end
|
| 179 |
+
|
| 180 |
+
SM->>DA: saveDocument(docId, resolved_content)
|
| 181 |
+
DA->>SB: UPDATE documents
|
| 182 |
+
SB-->>DA: Success
|
| 183 |
+
end
|
| 184 |
+
SM->>SM: showToast("تم مزامنة المستندات")
|
| 185 |
+
end
|
| 186 |
+
```
|
| 187 |
+
|
| 188 |
+
## Design Rationale
|
| 189 |
+
|
| 190 |
+
1. **Debounced Analysis**: 1-second debounce prevents API flooding during fast typing.
|
| 191 |
+
2. **Sequential NLP Pipeline**: Spelling → Grammar → Punctuation ensures each stage works on corrected text.
|
| 192 |
+
3. **Offline-First**: All changes saved to localStorage immediately; Supabase sync is eventual.
|
| 193 |
+
4. **Conflict Resolution**: Last-write-wins strategy with timestamp comparison.
|
docs/architecture/05-dataflow-diagram.md
ADDED
|
@@ -0,0 +1,135 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 05 — Data Flow Diagram (DFD)
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
This diagram traces how data flows through the BAYAN system — from user input to final storage, through all NLP processing stages.
|
| 6 |
+
|
| 7 |
+
## Level 0 — Context Diagram
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
graph LR
|
| 11 |
+
USER["👤 User"] -->|"Arabic Text"| BAYAN["🔵 BAYAN System"]
|
| 12 |
+
BAYAN -->|"Corrections · Summaries · Documents"| USER
|
| 13 |
+
BAYAN <-->|"Auth · Data"| SUPABASE["🗄️ Supabase"]
|
| 14 |
+
BAYAN <-->|"Models"| HUGGINGFACE["🤗 HuggingFace"]
|
| 15 |
+
```
|
| 16 |
+
|
| 17 |
+
## Level 1 — System DFD
|
| 18 |
+
|
| 19 |
+
```mermaid
|
| 20 |
+
graph TD
|
| 21 |
+
subgraph "Data Sources"
|
| 22 |
+
INPUT["📝 User Input Text"]
|
| 23 |
+
IMPORT["📥 Imported File<br/>(TXT / DOCX / PDF)"]
|
| 24 |
+
end
|
| 25 |
+
|
| 26 |
+
subgraph "1.0 Text Correction Pipeline"
|
| 27 |
+
SPELL["1.1 AraSpell<br/>Spelling Correction"]
|
| 28 |
+
GRAMMAR["1.2 Grammar Check<br/>Bayan Grammar"]
|
| 29 |
+
PUNCT["1.3 Punctuation<br/>PuncAra-v1"]
|
| 30 |
+
MERGE["1.4 Merge Results<br/>Offset Mapping"]
|
| 31 |
+
end
|
| 32 |
+
|
| 33 |
+
subgraph "2.0 Display Pipeline"
|
| 34 |
+
HIGHLIGHT["2.1 Error Highlighting<br/>renderer.js"]
|
| 35 |
+
SIDEBAR["2.2 Suggestions Panel<br/>ui.js"]
|
| 36 |
+
SCORE["2.3 Writing Score<br/>Calculation"]
|
| 37 |
+
STATS["2.4 Editor Stats<br/>Words · Chars · Sentences"]
|
| 38 |
+
end
|
| 39 |
+
|
| 40 |
+
subgraph "3.0 Summarization Pipeline"
|
| 41 |
+
SUMM_IN["3.1 Extract Clean Text"]
|
| 42 |
+
SUMM_MODEL["3.2 MBart Summarization"]
|
| 43 |
+
SUMM_OUT["3.3 Summary Display"]
|
| 44 |
+
end
|
| 45 |
+
|
| 46 |
+
subgraph "4.0 AutoComplete Pipeline"
|
| 47 |
+
AC_IN["4.1 Cursor Context"]
|
| 48 |
+
AC_MODEL["4.2 AutoComplete Model"]
|
| 49 |
+
AC_OUT["4.3 Dropdown Suggestions"]
|
| 50 |
+
end
|
| 51 |
+
|
| 52 |
+
subgraph "5.0 Storage"
|
| 53 |
+
LOCAL["5.1 localStorage<br/>Draft · Dismissed Words"]
|
| 54 |
+
DOCUMENTS_DB["5.2 Supabase: documents"]
|
| 55 |
+
SUMMARIES_DB["5.3 Supabase: summaries"]
|
| 56 |
+
SETTINGS_DB["5.4 Supabase: settings"]
|
| 57 |
+
PROFILES_DB["5.5 Supabase: profiles"]
|
| 58 |
+
end
|
| 59 |
+
|
| 60 |
+
subgraph "6.0 Export"
|
| 61 |
+
EXPORT_TXT["6.1 Export TXT"]
|
| 62 |
+
EXPORT_DOCX["6.2 Export DOCX"]
|
| 63 |
+
EXPORT_PDF["6.3 Export PDF"]
|
| 64 |
+
end
|
| 65 |
+
|
| 66 |
+
INPUT --> SPELL
|
| 67 |
+
IMPORT --> SPELL
|
| 68 |
+
SPELL -->|"Corrected tokens"| GRAMMAR
|
| 69 |
+
GRAMMAR -->|"Grammar-checked text"| PUNCT
|
| 70 |
+
PUNCT -->|"Punctuated text"| MERGE
|
| 71 |
+
MERGE -->|"All suggestions + offsets"| HIGHLIGHT
|
| 72 |
+
MERGE --> SIDEBAR
|
| 73 |
+
MERGE --> SCORE
|
| 74 |
+
INPUT --> STATS
|
| 75 |
+
|
| 76 |
+
INPUT --> SUMM_IN
|
| 77 |
+
SUMM_IN --> SUMM_MODEL
|
| 78 |
+
SUMM_MODEL --> SUMM_OUT
|
| 79 |
+
SUMM_OUT --> SUMMARIES_DB
|
| 80 |
+
|
| 81 |
+
INPUT --> AC_IN
|
| 82 |
+
AC_IN --> AC_MODEL
|
| 83 |
+
AC_MODEL --> AC_OUT
|
| 84 |
+
|
| 85 |
+
INPUT --> LOCAL
|
| 86 |
+
LOCAL -->|"Sync"| DOCUMENTS_DB
|
| 87 |
+
|
| 88 |
+
INPUT --> EXPORT_TXT
|
| 89 |
+
INPUT --> EXPORT_DOCX
|
| 90 |
+
INPUT --> EXPORT_PDF
|
| 91 |
+
|
| 92 |
+
style SPELL fill:#EF4444,color:#fff
|
| 93 |
+
style GRAMMAR fill:#F59E0B,color:#000
|
| 94 |
+
style PUNCT fill:#3B82F6,color:#fff
|
| 95 |
+
style SUMM_MODEL fill:#8B5CF6,color:#fff
|
| 96 |
+
style AC_MODEL fill:#10B981,color:#fff
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
## Data Stores Summary
|
| 100 |
+
|
| 101 |
+
| Store | Type | Data | Access Pattern |
|
| 102 |
+
|-------|------|------|----------------|
|
| 103 |
+
| `localStorage` | Client-side | Draft HTML, dismissed words, word goal, theme | Read/write on every input |
|
| 104 |
+
| `documents` | Supabase | id, user_id, title, content, timestamps | CRUD per user (RLS) |
|
| 105 |
+
| `summaries` | Supabase | id, document_id, user_id, original, summary | Append-mostly |
|
| 106 |
+
| `settings` | Supabase | id, user_id, preferences JSON | Read on login, write on change |
|
| 107 |
+
| `profiles` | Supabase | id, display_name, avatar_url, auth_provider | Auto-created on signup |
|
| 108 |
+
|
| 109 |
+
## Data Transformation Chain
|
| 110 |
+
|
| 111 |
+
```
|
| 112 |
+
Raw Arabic Text
|
| 113 |
+
│
|
| 114 |
+
├──→ AraSpell ──→ Corrected words (with alternatives)
|
| 115 |
+
│ │
|
| 116 |
+
│ ▼
|
| 117 |
+
│ Grammar Engine ──→ Grammar errors (with suggestions)
|
| 118 |
+
│ │
|
| 119 |
+
│ ▼
|
| 120 |
+
│ Punctuation ──→ Punctuation insertions
|
| 121 |
+
│ │
|
| 122 |
+
│ ▼
|
| 123 |
+
│ Offset Mapper ──→ Maps corrected offsets → original offsets
|
| 124 |
+
│ │
|
| 125 |
+
│ ▼
|
| 126 |
+
│ Unified Suggestions Array
|
| 127 |
+
│ │
|
| 128 |
+
│ ├──→ Error Spans (renderer.js)
|
| 129 |
+
│ ├──→ Sidebar Cards (ui.js)
|
| 130 |
+
│ └──→ Score Calculation
|
| 131 |
+
│
|
| 132 |
+
├──→ AutoComplete Model ──→ Top-K word suggestions
|
| 133 |
+
│
|
| 134 |
+
└──→ MBart Summarizer ──→ Compressed summary text
|
| 135 |
+
```
|
docs/architecture/06-deployment-diagram.md
ADDED
|
@@ -0,0 +1,130 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 06 — Deployment Diagram
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
BAYAN's final production deployment runs as a Dockerized Flask application on HuggingFace Spaces, with Supabase for database/auth and Google OAuth as an identity provider.
|
| 6 |
+
|
| 7 |
+
## Deployment Diagram
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
graph TB
|
| 11 |
+
subgraph "User Environment"
|
| 12 |
+
BROWSER["🌐 User Browser<br/>Chrome · Firefox · Safari"]
|
| 13 |
+
end
|
| 14 |
+
|
| 15 |
+
subgraph "GitHub"
|
| 16 |
+
REPO["📦 GitHub Repository<br/>mohamedatef24/BAYAN"]
|
| 17 |
+
CI["⚙️ GitHub Actions<br/>CI/CD Pipeline"]
|
| 18 |
+
end
|
| 19 |
+
|
| 20 |
+
subgraph "HuggingFace Spaces"
|
| 21 |
+
subgraph "Docker Container (python:3.12-slim)"
|
| 22 |
+
GUNICORN["Gunicorn<br/>WSGI Server<br/>1 Worker · Port 7860"]
|
| 23 |
+
|
| 24 |
+
subgraph "Flask Application"
|
| 25 |
+
FLASK_APP["Flask App<br/>CORS · Static Files"]
|
| 26 |
+
|
| 27 |
+
subgraph "Static Assets"
|
| 28 |
+
HTML["index.html"]
|
| 29 |
+
CSS_DIR["css/<br/>main.css · components.css"]
|
| 30 |
+
JS_DIR["js/<br/>28 JS modules"]
|
| 31 |
+
VENDOR["vendor/<br/>docx.min.js · jspdf"]
|
| 32 |
+
end
|
| 33 |
+
|
| 34 |
+
subgraph "API Endpoints"
|
| 35 |
+
EP1["GET / — SPA Entry"]
|
| 36 |
+
EP2["GET /api/health"]
|
| 37 |
+
EP3["POST /api/analyze"]
|
| 38 |
+
EP4["POST /api/spelling"]
|
| 39 |
+
EP5["POST /api/grammar"]
|
| 40 |
+
EP6["POST /api/punctuation"]
|
| 41 |
+
EP7["POST /api/summarize"]
|
| 42 |
+
EP8["POST /api/autocomplete"]
|
| 43 |
+
EP9["GET /api/debug/models"]
|
| 44 |
+
end
|
| 45 |
+
end
|
| 46 |
+
|
| 47 |
+
subgraph "NLP Models (Pre-cached)"
|
| 48 |
+
M1["AraSpell<br/>AraBERT Encoder-Decoder<br/>+ last_model.pt checkpoint<br/>~220MB"]
|
| 49 |
+
M2["Grammar Engine<br/>Rule-based + ML<br/>~50MB"]
|
| 50 |
+
M3["PuncAra-v1<br/>Punctuation Model<br/>~100MB"]
|
| 51 |
+
M4["AutoComplete<br/>Language Model<br/>~100MB"]
|
| 52 |
+
M5["Summarization<br/>MBart (float16)<br/>~600MB"]
|
| 53 |
+
end
|
| 54 |
+
|
| 55 |
+
ML_LOADER["ModelLoader<br/>Lazy Init · Singleton"]
|
| 56 |
+
end
|
| 57 |
+
end
|
| 58 |
+
|
| 59 |
+
subgraph "External Services"
|
| 60 |
+
SUPABASE["🗄️ Supabase<br/>PostgreSQL + Auth + RLS<br/>ap-southeast-1"]
|
| 61 |
+
GOOGLE["🔐 Google OAuth<br/>Identity Provider"]
|
| 62 |
+
HF_HUB["🤗 HuggingFace Hub<br/>Model Registry"]
|
| 63 |
+
end
|
| 64 |
+
|
| 65 |
+
BROWSER -->|"HTTPS"| GUNICORN
|
| 66 |
+
GUNICORN --> FLASK_APP
|
| 67 |
+
FLASK_APP --> EP1 & EP2 & EP3 & EP4 & EP5 & EP6 & EP7 & EP8 & EP9
|
| 68 |
+
EP3 & EP4 & EP5 & EP6 & EP7 & EP8 --> ML_LOADER
|
| 69 |
+
ML_LOADER --> M1 & M2 & M3 & M4 & M5
|
| 70 |
+
|
| 71 |
+
BROWSER -->|"Supabase JS SDK"| SUPABASE
|
| 72 |
+
BROWSER -->|"OAuth Redirect"| GOOGLE
|
| 73 |
+
GOOGLE -->|"Token"| SUPABASE
|
| 74 |
+
|
| 75 |
+
REPO -->|"Push to main"| CI
|
| 76 |
+
CI -->|"Deploy"| HF_HUB
|
| 77 |
+
|
| 78 |
+
style GUNICORN fill:#059669,color:#fff
|
| 79 |
+
style SUPABASE fill:#3B82F6,color:#fff
|
| 80 |
+
style GOOGLE fill:#DB4437,color:#fff
|
| 81 |
+
style ML_LOADER fill:#7C3AED,color:#fff
|
| 82 |
+
style HF_HUB fill:#FF9D00,color:#fff
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
## Container Specifications
|
| 86 |
+
|
| 87 |
+
| Parameter | Value |
|
| 88 |
+
|-----------|-------|
|
| 89 |
+
| **Base Image** | `python:3.12-slim` |
|
| 90 |
+
| **Port** | `7860` |
|
| 91 |
+
| **WSGI Server** | Gunicorn (1 worker, 120s timeout) |
|
| 92 |
+
| **PyTorch** | CPU-only (saves ~1.5GB vs CUDA) |
|
| 93 |
+
| **Total Model Size** | ~1.07 GB |
|
| 94 |
+
| **Estimated RAM** | ~2.5 GB (peak during inference) |
|
| 95 |
+
|
| 96 |
+
## Environment Variables
|
| 97 |
+
|
| 98 |
+
| Variable | Purpose | Source |
|
| 99 |
+
|----------|---------|--------|
|
| 100 |
+
| `SUPABASE_URL` | Database endpoint | HF Spaces Secrets |
|
| 101 |
+
| `SUPABASE_ANON_KEY` | Public API key | HF Spaces Secrets |
|
| 102 |
+
| `HF_API_TOKEN` | Remote inference fallback | HF Spaces Secrets |
|
| 103 |
+
| `SUMMARIZATION_REPO_ID` | Model repo path | Default: `bayan10/summarization-model` |
|
| 104 |
+
| `PORT` | Server port | Default: `7860` |
|
| 105 |
+
| `DEBUG` | Debug mode | Default: `False` |
|
| 106 |
+
|
| 107 |
+
## CI/CD Pipeline
|
| 108 |
+
|
| 109 |
+
```mermaid
|
| 110 |
+
graph LR
|
| 111 |
+
A["Developer Push<br/>to main"] --> B["GitHub Actions<br/>Triggered"]
|
| 112 |
+
B --> C["Lint & Validate<br/>Flask imports · Routes"]
|
| 113 |
+
C --> D["Build Script<br/>Inject Supabase creds"]
|
| 114 |
+
D --> E["Push to HF Spaces<br/>via git remote"]
|
| 115 |
+
E --> F["Docker Build<br/>on HF Spaces"]
|
| 116 |
+
F --> G["Pre-download Models<br/>During Build"]
|
| 117 |
+
G --> H["Container Start<br/>Gunicorn"]
|
| 118 |
+
H --> I["Health Check<br/>/api/health"]
|
| 119 |
+
|
| 120 |
+
style A fill:#4F46E5,color:#fff
|
| 121 |
+
style H fill:#059669,color:#fff
|
| 122 |
+
style I fill:#22C55E,color:#fff
|
| 123 |
+
```
|
| 124 |
+
|
| 125 |
+
## Scaling Considerations
|
| 126 |
+
|
| 127 |
+
- **Single Worker**: Minimizes RAM; ML models are not thread-safe.
|
| 128 |
+
- **Model Pre-caching**: Docker builds download models once; no runtime network needed.
|
| 129 |
+
- **HF Inference Fallback**: When `HF_API_TOKEN` is set, uses remote HF Inference API to avoid local RAM limits.
|
| 130 |
+
- **Float16 Models**: Summarization model loaded in half-precision to halve memory.
|
docs/architecture/07-database-schema.md
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 07 — Database Schema Diagram
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
BAYAN uses Supabase (PostgreSQL) with Row-Level Security (RLS). All tables enforce per-user data isolation via `auth.uid()`.
|
| 6 |
+
|
| 7 |
+
## Entity Relationship Diagram
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
erDiagram
|
| 11 |
+
AUTH_USERS {
|
| 12 |
+
uuid id PK
|
| 13 |
+
text email
|
| 14 |
+
boolean is_anonymous
|
| 15 |
+
jsonb raw_user_meta_data
|
| 16 |
+
jsonb raw_app_meta_data
|
| 17 |
+
timestamptz created_at
|
| 18 |
+
timestamptz updated_at
|
| 19 |
+
}
|
| 20 |
+
|
| 21 |
+
PROFILES {
|
| 22 |
+
uuid id PK "FK → auth.users(id)"
|
| 23 |
+
text display_name
|
| 24 |
+
text avatar_url
|
| 25 |
+
text auth_provider "anonymous | google"
|
| 26 |
+
timestamptz created_at
|
| 27 |
+
timestamptz updated_at
|
| 28 |
+
}
|
| 29 |
+
|
| 30 |
+
DOCUMENTS {
|
| 31 |
+
uuid id PK "gen_random_uuid()"
|
| 32 |
+
uuid user_id FK "→ auth.users(id)"
|
| 33 |
+
text title "Default: مستند جديد"
|
| 34 |
+
text content "HTML content"
|
| 35 |
+
timestamptz created_at
|
| 36 |
+
timestamptz updated_at
|
| 37 |
+
}
|
| 38 |
+
|
| 39 |
+
SUMMARIES {
|
| 40 |
+
uuid id PK "gen_random_uuid()"
|
| 41 |
+
uuid document_id FK "→ documents(id)"
|
| 42 |
+
uuid user_id FK "→ auth.users(id)"
|
| 43 |
+
text original_text
|
| 44 |
+
text summary_text
|
| 45 |
+
integer word_count
|
| 46 |
+
real compression_ratio
|
| 47 |
+
timestamptz created_at
|
| 48 |
+
}
|
| 49 |
+
|
| 50 |
+
SETTINGS {
|
| 51 |
+
uuid id PK "gen_random_uuid()"
|
| 52 |
+
uuid user_id FK "→ auth.users(id)"
|
| 53 |
+
jsonb preferences "theme, font, size, etc."
|
| 54 |
+
timestamptz created_at
|
| 55 |
+
timestamptz updated_at
|
| 56 |
+
}
|
| 57 |
+
|
| 58 |
+
AUTH_USERS ||--|| PROFILES : "auto-created on signup"
|
| 59 |
+
AUTH_USERS ||--o{ DOCUMENTS : "owns"
|
| 60 |
+
AUTH_USERS ||--o| SETTINGS : "has"
|
| 61 |
+
DOCUMENTS ||--o{ SUMMARIES : "has"
|
| 62 |
+
AUTH_USERS ||--o{ SUMMARIES : "owns"
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
## Table Details
|
| 66 |
+
|
| 67 |
+
### profiles
|
| 68 |
+
|
| 69 |
+
| Column | Type | Constraints | Description |
|
| 70 |
+
|--------|------|-------------|-------------|
|
| 71 |
+
| `id` | `uuid` | PK, FK → auth.users | User ID from Supabase Auth |
|
| 72 |
+
| `display_name` | `text` | nullable | Display name (from Google or "ضيف") |
|
| 73 |
+
| `avatar_url` | `text` | nullable | Google profile picture URL |
|
| 74 |
+
| `auth_provider` | `text` | NOT NULL, default 'anonymous' | `anonymous` or `google` |
|
| 75 |
+
| `created_at` | `timestamptz` | NOT NULL, default now() | Account creation time |
|
| 76 |
+
| `updated_at` | `timestamptz` | NOT NULL, default now() | Last profile update |
|
| 77 |
+
|
| 78 |
+
### documents
|
| 79 |
+
|
| 80 |
+
| Column | Type | Constraints | Description |
|
| 81 |
+
|--------|------|-------------|-------------|
|
| 82 |
+
| `id` | `uuid` | PK, default gen_random_uuid() | Document unique ID |
|
| 83 |
+
| `user_id` | `uuid` | FK → auth.users, NOT NULL | Owner |
|
| 84 |
+
| `title` | `text` | default 'مستند جديد' | Document title |
|
| 85 |
+
| `content` | `text` | nullable | HTML content from editor |
|
| 86 |
+
| `created_at` | `timestamptz` | NOT NULL, default now() | Creation time |
|
| 87 |
+
| `updated_at` | `timestamptz` | NOT NULL, default now() | Last save time |
|
| 88 |
+
|
| 89 |
+
### summaries
|
| 90 |
+
|
| 91 |
+
| Column | Type | Constraints | Description |
|
| 92 |
+
|--------|------|-------------|-------------|
|
| 93 |
+
| `id` | `uuid` | PK, default gen_random_uuid() | Summary unique ID |
|
| 94 |
+
| `document_id` | `uuid` | FK → documents(id) | Parent document |
|
| 95 |
+
| `user_id` | `uuid` | FK → auth.users, NOT NULL | Owner |
|
| 96 |
+
| `original_text` | `text` | NOT NULL | Source text that was summarized |
|
| 97 |
+
| `summary_text` | `text` | NOT NULL | Generated summary |
|
| 98 |
+
| `word_count` | `integer` | nullable | Summary word count |
|
| 99 |
+
| `compression_ratio` | `real` | nullable | Compression percentage |
|
| 100 |
+
| `created_at` | `timestamptz` | NOT NULL, default now() | Generation time |
|
| 101 |
+
|
| 102 |
+
### settings
|
| 103 |
+
|
| 104 |
+
| Column | Type | Constraints | Description |
|
| 105 |
+
|--------|------|-------------|-------------|
|
| 106 |
+
| `id` | `uuid` | PK, default gen_random_uuid() | Setting record ID |
|
| 107 |
+
| `user_id` | `uuid` | FK → auth.users, UNIQUE | Owner (one per user) |
|
| 108 |
+
| `preferences` | `jsonb` | NOT NULL, default '{}' | All user preferences |
|
| 109 |
+
| `created_at` | `timestamptz` | NOT NULL, default now() | Creation time |
|
| 110 |
+
| `updated_at` | `timestamptz` | NOT NULL, default now() | Last update time |
|
| 111 |
+
|
| 112 |
+
## Row Level Security (RLS) Policies
|
| 113 |
+
|
| 114 |
+
All tables have RLS enabled. Policies follow the pattern:
|
| 115 |
+
|
| 116 |
+
```sql
|
| 117 |
+
-- SELECT: Users can only read their own data
|
| 118 |
+
CREATE POLICY "select_own" ON table
|
| 119 |
+
FOR SELECT USING (auth.uid() = user_id);
|
| 120 |
+
|
| 121 |
+
-- INSERT: Users can only insert their own data
|
| 122 |
+
CREATE POLICY "insert_own" ON table
|
| 123 |
+
FOR INSERT WITH CHECK (auth.uid() = user_id);
|
| 124 |
+
|
| 125 |
+
-- UPDATE: Users can only update their own data
|
| 126 |
+
CREATE POLICY "update_own" ON table
|
| 127 |
+
FOR UPDATE USING (auth.uid() = user_id);
|
| 128 |
+
|
| 129 |
+
-- DELETE: Users can only delete their own data
|
| 130 |
+
CREATE POLICY "delete_own" ON table
|
| 131 |
+
FOR DELETE USING (auth.uid() = user_id);
|
| 132 |
+
```
|
| 133 |
+
|
| 134 |
+
## Database Triggers
|
| 135 |
+
|
| 136 |
+
| Trigger | Event | Function | Purpose |
|
| 137 |
+
|---------|-------|----------|---------|
|
| 138 |
+
| `on_auth_user_created` | `AFTER INSERT ON auth.users` | `handle_new_user()` | Auto-create profile on signup |
|
| 139 |
+
| `on_auth_user_updated` | `AFTER UPDATE ON auth.users` | `handle_user_updated()` | Update profile when linking Google |
|
| 140 |
+
|
| 141 |
+
## Client-Side Storage (localStorage)
|
| 142 |
+
|
| 143 |
+
| Key | Type | Purpose |
|
| 144 |
+
|-----|------|---------|
|
| 145 |
+
| `bayan_editor_draft` | `string (HTML)` | Unsaved editor content |
|
| 146 |
+
| `bayan_dismissed_words` | `JSON array` | Words marked "keep as-is" |
|
| 147 |
+
| `bayan_word_goal` | `number` | Word count target |
|
| 148 |
+
| `bayan_theme` | `string` | "dark" or "light" |
|
docs/architecture/08-nlp-pipeline-diagram.md
ADDED
|
@@ -0,0 +1,198 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 08 — NLP Pipeline Diagram (Final Production State)
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
The BAYAN NLP layer consists of three independent pipelines: **Text Correction** (sequential), **AutoComplete** (parallel), and **Summarization** (independent). All five models are fully deployed and operational.
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## Main Correction Pipeline
|
| 10 |
+
|
| 11 |
+
```mermaid
|
| 12 |
+
graph TD
|
| 13 |
+
subgraph "Input"
|
| 14 |
+
TEXT["📝 User Input Text<br/>(Arabic, up to 5000 chars)"]
|
| 15 |
+
end
|
| 16 |
+
|
| 17 |
+
subgraph "Stage 1: Spelling Correction"
|
| 18 |
+
TOKENIZE["1.1 Word Tokenization<br/>Split into Arabic tokens"]
|
| 19 |
+
SPELL_CHECK["1.2 AraSpell Check<br/>AraBERT Encoder-Decoder"]
|
| 20 |
+
ALT_GEN["1.3 Alternative Generation<br/>Top-K Candidates"]
|
| 21 |
+
SPELL_OUT["1.4 Spelling Corrections<br/>{original, correction, alternatives, start, end}"]
|
| 22 |
+
end
|
| 23 |
+
|
| 24 |
+
subgraph "Stage 2: Grammar Checking"
|
| 25 |
+
CORRECTED_TEXT["2.0 Spelling-Corrected Text"]
|
| 26 |
+
GRAM_RULES["2.1 Grammar Rules Engine<br/>Subject-Verb Agreement<br/>Case Marking<br/>Definiteness"]
|
| 27 |
+
GRAM_ML["2.2 ML Grammar Model<br/>Contextual Analysis"]
|
| 28 |
+
GRAM_OUT["2.3 Grammar Errors<br/>{original, correction, type: grammar}"]
|
| 29 |
+
end
|
| 30 |
+
|
| 31 |
+
subgraph "Stage 3: Punctuation Restoration"
|
| 32 |
+
CLEAN_TEXT["3.0 Grammar-Corrected Text"]
|
| 33 |
+
PUNCT_MODEL["3.1 PuncAra-v1<br/>Sequence Labeling"]
|
| 34 |
+
PUNCT_OUT["3.2 Punctuation Suggestions<br/>{position, punctuation_mark}"]
|
| 35 |
+
end
|
| 36 |
+
|
| 37 |
+
subgraph "Stage 4: Result Merging"
|
| 38 |
+
OFFSET_MAP["4.1 Offset Mapper<br/>Map corrected → original positions"]
|
| 39 |
+
MERGE["4.2 Merge All Suggestions<br/>spelling + grammar + punctuation"]
|
| 40 |
+
SCORE["4.3 Writing Score Calculation<br/>100 - (errors × weight)"]
|
| 41 |
+
end
|
| 42 |
+
|
| 43 |
+
subgraph "Output"
|
| 44 |
+
RESULT["📊 Unified Response<br/>{suggestions[], score, counts}"]
|
| 45 |
+
end
|
| 46 |
+
|
| 47 |
+
TEXT --> TOKENIZE
|
| 48 |
+
TOKENIZE --> SPELL_CHECK
|
| 49 |
+
SPELL_CHECK --> ALT_GEN
|
| 50 |
+
ALT_GEN --> SPELL_OUT
|
| 51 |
+
|
| 52 |
+
SPELL_OUT --> CORRECTED_TEXT
|
| 53 |
+
CORRECTED_TEXT --> GRAM_RULES
|
| 54 |
+
CORRECTED_TEXT --> GRAM_ML
|
| 55 |
+
GRAM_RULES --> GRAM_OUT
|
| 56 |
+
GRAM_ML --> GRAM_OUT
|
| 57 |
+
|
| 58 |
+
GRAM_OUT --> CLEAN_TEXT
|
| 59 |
+
CLEAN_TEXT --> PUNCT_MODEL
|
| 60 |
+
PUNCT_MODEL --> PUNCT_OUT
|
| 61 |
+
|
| 62 |
+
SPELL_OUT --> OFFSET_MAP
|
| 63 |
+
GRAM_OUT --> OFFSET_MAP
|
| 64 |
+
PUNCT_OUT --> OFFSET_MAP
|
| 65 |
+
OFFSET_MAP --> MERGE
|
| 66 |
+
MERGE --> SCORE
|
| 67 |
+
SCORE --> RESULT
|
| 68 |
+
|
| 69 |
+
style SPELL_CHECK fill:#EF4444,color:#fff
|
| 70 |
+
style GRAM_RULES fill:#F59E0B,color:#000
|
| 71 |
+
style PUNCT_MODEL fill:#3B82F6,color:#fff
|
| 72 |
+
style SCORE fill:#22C55E,color:#fff
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
---
|
| 76 |
+
|
| 77 |
+
## Smart Dependency Logic
|
| 78 |
+
|
| 79 |
+
```mermaid
|
| 80 |
+
graph LR
|
| 81 |
+
subgraph "Token-Level Pipeline"
|
| 82 |
+
T1["Token 1<br/>'الكتابه'"]
|
| 83 |
+
T2["Token 2<br/>'الصحيحه'"]
|
| 84 |
+
T3["Token 3<br/>'مهمة'"]
|
| 85 |
+
T4["Token 4<br/>'جدا'"]
|
| 86 |
+
end
|
| 87 |
+
|
| 88 |
+
subgraph "AraSpell"
|
| 89 |
+
S1["❌ الكتابه → الكتابة"]
|
| 90 |
+
S2["❌ الصحيحه → الصحيحة"]
|
| 91 |
+
S3["✅ مهمة (clean)"]
|
| 92 |
+
S4["✅ جدا (clean)"]
|
| 93 |
+
end
|
| 94 |
+
|
| 95 |
+
subgraph "Grammar Engine"
|
| 96 |
+
G3["✅ مهمة<br/>(processed immediately)"]
|
| 97 |
+
G4["✅ جداً<br/>(processed immediately)"]
|
| 98 |
+
G1["⏳ الكتابة<br/>(waits for correction)"]
|
| 99 |
+
G2["⏳ الصحيحة<br/>(waits for correction)"]
|
| 100 |
+
end
|
| 101 |
+
|
| 102 |
+
subgraph "Punctuation"
|
| 103 |
+
P["PuncAra processes<br/>final corrected sentence"]
|
| 104 |
+
end
|
| 105 |
+
|
| 106 |
+
T1 --> S1
|
| 107 |
+
T2 --> S2
|
| 108 |
+
T3 --> S3
|
| 109 |
+
T4 --> S4
|
| 110 |
+
|
| 111 |
+
S3 --> G3
|
| 112 |
+
S4 --> G4
|
| 113 |
+
S1 -.->|"after correction"| G1
|
| 114 |
+
S2 -.->|"after correction"| G2
|
| 115 |
+
|
| 116 |
+
G1 & G2 & G3 & G4 --> P
|
| 117 |
+
|
| 118 |
+
style S1 fill:#EF4444,color:#fff
|
| 119 |
+
style S2 fill:#EF4444,color:#fff
|
| 120 |
+
style S3 fill:#22C55E,color:#fff
|
| 121 |
+
style S4 fill:#22C55E,color:#fff
|
| 122 |
+
```
|
| 123 |
+
|
| 124 |
+
### Dependency Rules
|
| 125 |
+
|
| 126 |
+
1. **Clean tokens** bypass the spelling wait and are sent to grammar immediately.
|
| 127 |
+
2. **Misspelled tokens** are corrected first; grammar waits only for those specific tokens.
|
| 128 |
+
3. **Punctuation** executes last on the fully corrected sentence.
|
| 129 |
+
4. **Offset mapping** traces every change back to the original text positions.
|
| 130 |
+
|
| 131 |
+
---
|
| 132 |
+
|
| 133 |
+
## AutoComplete Pipeline
|
| 134 |
+
|
| 135 |
+
```mermaid
|
| 136 |
+
graph TD
|
| 137 |
+
TYPING["⌨️ User Typing<br/>Partial word detected"]
|
| 138 |
+
CONTEXT["Extract Context<br/>Last 50 tokens + cursor position"]
|
| 139 |
+
AC_MODEL["AutoComplete Model<br/>Language Model Inference"]
|
| 140 |
+
RANKING["Suggestion Ranking<br/>Score + Frequency + Relevance"]
|
| 141 |
+
FILTER["Arabic-Only Filter<br/>Remove non-Arabic candidates"]
|
| 142 |
+
DROPDOWN["📋 Dropdown Display<br/>Top 3-5 suggestions"]
|
| 143 |
+
INSERT["Insert Selected<br/>Replace partial word"]
|
| 144 |
+
|
| 145 |
+
TYPING --> CONTEXT
|
| 146 |
+
CONTEXT --> AC_MODEL
|
| 147 |
+
AC_MODEL --> RANKING
|
| 148 |
+
RANKING --> FILTER
|
| 149 |
+
FILTER --> DROPDOWN
|
| 150 |
+
DROPDOWN -->|"User selects"| INSERT
|
| 151 |
+
|
| 152 |
+
style AC_MODEL fill:#10B981,color:#fff
|
| 153 |
+
```
|
| 154 |
+
|
| 155 |
+
---
|
| 156 |
+
|
| 157 |
+
## Summarization Pipeline
|
| 158 |
+
|
| 159 |
+
```mermaid
|
| 160 |
+
graph TD
|
| 161 |
+
SOURCE["📄 Source Text<br/>(from editor or document)"]
|
| 162 |
+
VALIDATE["Validate<br/>≥10 chars, ≤5000 chars"]
|
| 163 |
+
TOKENIZE["MBart Tokenizer<br/>src_lang=ar_AR"]
|
| 164 |
+
GENERATE["MBart Generation<br/>beam_search, max_tokens=512"]
|
| 165 |
+
DECODE["Decode Output<br/>skip_special_tokens"]
|
| 166 |
+
METRICS["Calculate Metrics<br/>word_count, compression_ratio"]
|
| 167 |
+
DISPLAY["📊 Display Summary<br/>Summary tab UI"]
|
| 168 |
+
STORE["💾 Save to Supabase<br/>summaries table"]
|
| 169 |
+
|
| 170 |
+
SOURCE --> VALIDATE
|
| 171 |
+
VALIDATE --> TOKENIZE
|
| 172 |
+
TOKENIZE --> GENERATE
|
| 173 |
+
GENERATE --> DECODE
|
| 174 |
+
DECODE --> METRICS
|
| 175 |
+
METRICS --> DISPLAY
|
| 176 |
+
METRICS --> STORE
|
| 177 |
+
|
| 178 |
+
style GENERATE fill:#8B5CF6,color:#fff
|
| 179 |
+
```
|
| 180 |
+
|
| 181 |
+
---
|
| 182 |
+
|
| 183 |
+
## Models Reference Table
|
| 184 |
+
|
| 185 |
+
| Model | Architecture | Size | Input | Output |
|
| 186 |
+
|-------|-------------|------|-------|--------|
|
| 187 |
+
| **AraSpell** | AraBERT Encoder-Decoder + Checkpoint | ~220MB | Misspelled word | Corrected word + alternatives |
|
| 188 |
+
| **Grammar** | Rule Engine + ML Classifier | ~50MB | Corrected text | Grammar errors + suggestions |
|
| 189 |
+
| **PuncAra-v1** | Sequence Labeling Model | ~100MB | Unpunctuated text | Punctuated text |
|
| 190 |
+
| **AutoComplete** | Language Model | ~100MB | Text prefix | Next word candidates |
|
| 191 |
+
| **Summarization** | MBart (float16) | ~600MB | Arabic text | Compressed summary |
|
| 192 |
+
|
| 193 |
+
## Design Rationale
|
| 194 |
+
|
| 195 |
+
1. **Sequential Pipeline**: Spelling must run first because grammar and punctuation analysis on misspelled words produces false positives.
|
| 196 |
+
2. **Offset Mapping**: Critical for mapping corrected positions back to original text (where the user sees highlights).
|
| 197 |
+
3. **HF Inference Fallback**: When RAM is limited, the system can route to HuggingFace Inference API remotely.
|
| 198 |
+
4. **Float16**: Summarization model uses half-precision to halve memory footprint without quality loss.
|
docs/architecture/09-sync-engine-diagram.md
ADDED
|
@@ -0,0 +1,179 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 09 — Sync Engine Diagram
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
The Sync Engine ensures documents are saved reliably, supporting offline editing with automatic recovery and conflict resolution when connectivity is restored.
|
| 6 |
+
|
| 7 |
+
## Sync Architecture
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
graph TD
|
| 11 |
+
subgraph "Editor Layer"
|
| 12 |
+
EDITOR["📝 Editor<br/>contenteditable div"]
|
| 13 |
+
INPUT["User Input Event"]
|
| 14 |
+
DRAFT["💾 localStorage Draft<br/>Immediate save"]
|
| 15 |
+
end
|
| 16 |
+
|
| 17 |
+
subgraph "Sync Manager (sync-manager.js)"
|
| 18 |
+
DEBOUNCE["⏱ Debounce Timer<br/>2000ms"]
|
| 19 |
+
ONLINE_CHECK["📶 Online Status<br/>navigator.onLine"]
|
| 20 |
+
SCHEDULE["scheduleSave()"]
|
| 21 |
+
SYNC_NOW["syncNow()"]
|
| 22 |
+
end
|
| 23 |
+
|
| 24 |
+
subgraph "Sync Queue (sync-queue.js)"
|
| 25 |
+
QUEUE["📋 Operation Queue<br/>[{type, docId, content, timestamp}]"]
|
| 26 |
+
ENQUEUE["enqueue()"]
|
| 27 |
+
DEQUEUE["dequeue()"]
|
| 28 |
+
FLUSH["flush()"]
|
| 29 |
+
end
|
| 30 |
+
|
| 31 |
+
subgraph "Conflict Resolver (sync-resolver.js)"
|
| 32 |
+
RESOLVE["resolve(local, remote)"]
|
| 33 |
+
LWW["lastWriteWins()"]
|
| 34 |
+
MERGE["mergeConflict()"]
|
| 35 |
+
end
|
| 36 |
+
|
| 37 |
+
subgraph "Cloud Storage"
|
| 38 |
+
DOC_API["documents-api.js"]
|
| 39 |
+
SUPABASE["🗄️ Supabase<br/>documents table"]
|
| 40 |
+
end
|
| 41 |
+
|
| 42 |
+
subgraph "UI Feedback"
|
| 43 |
+
TOAST["🔔 Toast Notification"]
|
| 44 |
+
STATUS["Auto-save Status<br/>'تم الحفظ' / 'سيتم الحفظ'"]
|
| 45 |
+
end
|
| 46 |
+
|
| 47 |
+
INPUT --> EDITOR
|
| 48 |
+
EDITOR --> DRAFT
|
| 49 |
+
EDITOR --> SCHEDULE
|
| 50 |
+
|
| 51 |
+
SCHEDULE --> DEBOUNCE
|
| 52 |
+
DEBOUNCE --> ONLINE_CHECK
|
| 53 |
+
|
| 54 |
+
ONLINE_CHECK -->|"Online"| SYNC_NOW
|
| 55 |
+
ONLINE_CHECK -->|"Offline"| ENQUEUE
|
| 56 |
+
ENQUEUE --> QUEUE
|
| 57 |
+
|
| 58 |
+
SYNC_NOW --> DOC_API
|
| 59 |
+
DOC_API --> SUPABASE
|
| 60 |
+
SUPABASE --> STATUS
|
| 61 |
+
|
| 62 |
+
QUEUE -->|"Back Online"| FLUSH
|
| 63 |
+
FLUSH --> DEQUEUE
|
| 64 |
+
DEQUEUE --> RESOLVE
|
| 65 |
+
RESOLVE --> LWW
|
| 66 |
+
RESOLVE --> MERGE
|
| 67 |
+
LWW --> DOC_API
|
| 68 |
+
MERGE --> DOC_API
|
| 69 |
+
|
| 70 |
+
DOC_API --> TOAST
|
| 71 |
+
|
| 72 |
+
style DRAFT fill:#F59E0B,color:#000
|
| 73 |
+
style QUEUE fill:#EF4444,color:#fff
|
| 74 |
+
style SUPABASE fill:#3B82F6,color:#fff
|
| 75 |
+
style RESOLVE fill:#8B5CF6,color:#fff
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
## State Machine
|
| 79 |
+
|
| 80 |
+
```mermaid
|
| 81 |
+
stateDiagram-v2
|
| 82 |
+
[*] --> Idle
|
| 83 |
+
|
| 84 |
+
Idle --> Editing : User types
|
| 85 |
+
Editing --> Debouncing : Input stops
|
| 86 |
+
Debouncing --> Editing : More input
|
| 87 |
+
Debouncing --> CheckOnline : Timer expires (2s)
|
| 88 |
+
|
| 89 |
+
CheckOnline --> Saving : Online
|
| 90 |
+
CheckOnline --> Queued : Offline
|
| 91 |
+
|
| 92 |
+
Saving --> Saved : Success
|
| 93 |
+
Saving --> Queued : Network error
|
| 94 |
+
Saved --> Idle : Status shown
|
| 95 |
+
|
| 96 |
+
Queued --> Flushing : Connection restored
|
| 97 |
+
Flushing --> ConflictCheck : Load remote
|
| 98 |
+
ConflictCheck --> Resolved : No conflict
|
| 99 |
+
ConflictCheck --> Merging : Conflict detected
|
| 100 |
+
Merging --> Resolved : Last-write-wins
|
| 101 |
+
Resolved --> Saving : Save merged
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
## Module API Reference
|
| 105 |
+
|
| 106 |
+
### SyncManager (`sync-manager.js`)
|
| 107 |
+
|
| 108 |
+
```javascript
|
| 109 |
+
// Initialize sync engine
|
| 110 |
+
initSync()
|
| 111 |
+
|
| 112 |
+
// Schedule a debounced save
|
| 113 |
+
scheduleSave()
|
| 114 |
+
|
| 115 |
+
// Immediately sync current document
|
| 116 |
+
async syncNow() → Promise<boolean>
|
| 117 |
+
|
| 118 |
+
// Handle browser online event
|
| 119 |
+
handleOnline()
|
| 120 |
+
|
| 121 |
+
// Handle browser offline event
|
| 122 |
+
handleOffline()
|
| 123 |
+
```
|
| 124 |
+
|
| 125 |
+
### SyncQueue (`sync-queue.js`)
|
| 126 |
+
|
| 127 |
+
```javascript
|
| 128 |
+
// Add operation to queue
|
| 129 |
+
enqueue({ type: 'save', docId: string, content: string })
|
| 130 |
+
|
| 131 |
+
// Remove and return next operation
|
| 132 |
+
dequeue() → Operation | null
|
| 133 |
+
|
| 134 |
+
// Check if queue is empty
|
| 135 |
+
isEmpty() → boolean
|
| 136 |
+
|
| 137 |
+
// Process all queued operations
|
| 138 |
+
async flush() → Promise<void>
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
### SyncResolver (`sync-resolver.js`)
|
| 142 |
+
|
| 143 |
+
```javascript
|
| 144 |
+
// Resolve local vs remote versions
|
| 145 |
+
resolve(local: Document, remote: Document) → ResolvedDocument
|
| 146 |
+
|
| 147 |
+
// Timestamp-based resolution
|
| 148 |
+
lastWriteWins(a: Document, b: Document) → Document
|
| 149 |
+
|
| 150 |
+
// Content merge (future: OT-based)
|
| 151 |
+
mergeConflict(localContent: string, remoteContent: string) → string
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
## Conflict Resolution Strategy
|
| 155 |
+
|
| 156 |
+
```mermaid
|
| 157 |
+
graph TD
|
| 158 |
+
A["Local Change<br/>timestamp: T1"] --> C{Compare Timestamps}
|
| 159 |
+
B["Remote Change<br/>timestamp: T2"] --> C
|
| 160 |
+
|
| 161 |
+
C -->|"T1 > T2"| D["Use Local<br/>(More recent)"]
|
| 162 |
+
C -->|"T2 > T1"| E["Use Remote<br/>(More recent)"]
|
| 163 |
+
C -->|"T1 = T2"| F["Content Diff"]
|
| 164 |
+
|
| 165 |
+
F -->|"Same content"| G["No Action"]
|
| 166 |
+
F -->|"Different"| H["Last Write Wins<br/>(Remote preferred)"]
|
| 167 |
+
|
| 168 |
+
style D fill:#22C55E,color:#fff
|
| 169 |
+
style E fill:#3B82F6,color:#fff
|
| 170 |
+
style H fill:#F59E0B,color:#000
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
+
## Design Rationale
|
| 174 |
+
|
| 175 |
+
1. **localStorage First**: Every keystroke saves to localStorage — zero data loss on crash/close.
|
| 176 |
+
2. **2-Second Debounce**: Prevents excessive Supabase writes during fast typing.
|
| 177 |
+
3. **Queue-Based Offline**: Operations queue in memory; flushed when connectivity returns.
|
| 178 |
+
4. **Last-Write-Wins**: Simple, predictable resolution — the most recent change wins.
|
| 179 |
+
5. **Browser Events**: `online`/`offline` events trigger queue flush and status updates.
|
docs/architecture/10-project-dependency-map.md
ADDED
|
@@ -0,0 +1,195 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 10 — Project Dependency Map
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
This diagram shows the dependency relationships between all BAYAN modules, highlighting critical paths, internal dependencies, and external services.
|
| 6 |
+
|
| 7 |
+
## Full Dependency Graph
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
graph TB
|
| 11 |
+
subgraph "Frontend Dependencies"
|
| 12 |
+
direction TB
|
| 13 |
+
INDEX["index.html<br/>(SPA Entry)"]
|
| 14 |
+
|
| 15 |
+
subgraph "Core"
|
| 16 |
+
EDITOR["editor.js"]
|
| 17 |
+
RENDERER["renderer.js"]
|
| 18 |
+
SELECTION["selection.js"]
|
| 19 |
+
FORMAT["format.js"]
|
| 20 |
+
end
|
| 21 |
+
|
| 22 |
+
subgraph "Auth"
|
| 23 |
+
AUTH["auth.js"]
|
| 24 |
+
AUTH_UI["auth-ui.js"]
|
| 25 |
+
SESSION["session.js"]
|
| 26 |
+
CLIENT["client.js"]
|
| 27 |
+
CONFIG["config.js"]
|
| 28 |
+
end
|
| 29 |
+
|
| 30 |
+
subgraph "Data"
|
| 31 |
+
DOC_API["documents-api.js"]
|
| 32 |
+
DOC_UI["documents-ui.js"]
|
| 33 |
+
DOC_STATE["documents-state.js"]
|
| 34 |
+
SUMM_API["summaries-api.js"]
|
| 35 |
+
SUMM_UI["summaries-ui.js"]
|
| 36 |
+
SET_API["settings-api.js"]
|
| 37 |
+
SET_SYNC["settings-sync.js"]
|
| 38 |
+
end
|
| 39 |
+
|
| 40 |
+
subgraph "Sync"
|
| 41 |
+
SYNC_MGR["sync-manager.js"]
|
| 42 |
+
SYNC_Q["sync-queue.js"]
|
| 43 |
+
SYNC_RES["sync-resolver.js"]
|
| 44 |
+
end
|
| 45 |
+
|
| 46 |
+
subgraph "UI"
|
| 47 |
+
UI["ui.js"]
|
| 48 |
+
THEME["theme.js"]
|
| 49 |
+
API["api.js"]
|
| 50 |
+
end
|
| 51 |
+
|
| 52 |
+
subgraph "Vendor"
|
| 53 |
+
DOCX["docx.min.js"]
|
| 54 |
+
JSPDF["jspdf.umd.min.js"]
|
| 55 |
+
SUPABASE_SDK["supabase-js (CDN)"]
|
| 56 |
+
end
|
| 57 |
+
end
|
| 58 |
+
|
| 59 |
+
subgraph "Backend Dependencies"
|
| 60 |
+
APP["app.py (Flask)"]
|
| 61 |
+
MODEL_LOADER["model_loader.py"]
|
| 62 |
+
HF_INF["hf_inference.py"]
|
| 63 |
+
|
| 64 |
+
subgraph "NLP"
|
| 65 |
+
ARASPELL_SVC["araspell_service.py"]
|
| 66 |
+
ARASPELL_RULES["araspell_rules.py"]
|
| 67 |
+
GRAMMAR_SVC["grammar_service.py"]
|
| 68 |
+
end
|
| 69 |
+
end
|
| 70 |
+
|
| 71 |
+
subgraph "External Services"
|
| 72 |
+
SUPABASE["Supabase<br/>Auth + PostgreSQL"]
|
| 73 |
+
GOOGLE["Google OAuth"]
|
| 74 |
+
HF_HUB["HuggingFace Hub"]
|
| 75 |
+
HF_API_EXT["HF Inference API"]
|
| 76 |
+
end
|
| 77 |
+
|
| 78 |
+
subgraph "Infrastructure"
|
| 79 |
+
DOCKER["Docker"]
|
| 80 |
+
GUNICORN["Gunicorn"]
|
| 81 |
+
GH_ACTIONS["GitHub Actions"]
|
| 82 |
+
end
|
| 83 |
+
|
| 84 |
+
subgraph "Python Packages"
|
| 85 |
+
FLASK["Flask + CORS"]
|
| 86 |
+
TORCH["PyTorch (CPU)"]
|
| 87 |
+
TRANSFORMERS["Transformers"]
|
| 88 |
+
HF_HUB_PKG["huggingface_hub"]
|
| 89 |
+
end
|
| 90 |
+
|
| 91 |
+
%% Frontend Internal Dependencies
|
| 92 |
+
INDEX --> EDITOR & RENDERER & SELECTION & FORMAT
|
| 93 |
+
INDEX --> AUTH & AUTH_UI & SESSION & CLIENT
|
| 94 |
+
INDEX --> DOC_API & DOC_UI & SUMM_API & SUMM_UI
|
| 95 |
+
INDEX --> SYNC_MGR & UI & THEME & API
|
| 96 |
+
|
| 97 |
+
EDITOR --> RENDERER
|
| 98 |
+
EDITOR --> SELECTION
|
| 99 |
+
EDITOR --> API
|
| 100 |
+
EDITOR --> UI
|
| 101 |
+
|
| 102 |
+
DOC_UI --> DOC_API
|
| 103 |
+
DOC_UI --> DOC_STATE
|
| 104 |
+
SUMM_UI --> SUMM_API
|
| 105 |
+
SET_SYNC --> SET_API
|
| 106 |
+
|
| 107 |
+
SYNC_MGR --> SYNC_Q
|
| 108 |
+
SYNC_MGR --> SYNC_RES
|
| 109 |
+
SYNC_MGR --> DOC_API
|
| 110 |
+
|
| 111 |
+
AUTH --> CLIENT
|
| 112 |
+
AUTH --> SESSION
|
| 113 |
+
AUTH_UI --> AUTH
|
| 114 |
+
CLIENT --> CONFIG
|
| 115 |
+
CLIENT --> SUPABASE_SDK
|
| 116 |
+
|
| 117 |
+
DOC_API --> CLIENT
|
| 118 |
+
SUMM_API --> CLIENT
|
| 119 |
+
SET_API --> CLIENT
|
| 120 |
+
|
| 121 |
+
DOC_UI --> DOCX
|
| 122 |
+
DOC_UI --> JSPDF
|
| 123 |
+
|
| 124 |
+
%% Backend Internal Dependencies
|
| 125 |
+
APP --> MODEL_LOADER
|
| 126 |
+
APP --> HF_INF
|
| 127 |
+
APP --> ARASPELL_SVC
|
| 128 |
+
|
| 129 |
+
ARASPELL_SVC --> ARASPELL_RULES
|
| 130 |
+
MODEL_LOADER --> TRANSFORMERS
|
| 131 |
+
MODEL_LOADER --> TORCH
|
| 132 |
+
MODEL_LOADER --> HF_HUB_PKG
|
| 133 |
+
|
| 134 |
+
HF_INF --> HF_API_EXT
|
| 135 |
+
|
| 136 |
+
%% External Dependencies
|
| 137 |
+
CLIENT --> SUPABASE
|
| 138 |
+
AUTH --> GOOGLE
|
| 139 |
+
MODEL_LOADER --> HF_HUB
|
| 140 |
+
DOCKER --> GUNICORN
|
| 141 |
+
GUNICORN --> APP
|
| 142 |
+
GH_ACTIONS --> DOCKER
|
| 143 |
+
|
| 144 |
+
%% Styling
|
| 145 |
+
style APP fill:#059669,color:#fff
|
| 146 |
+
style SUPABASE fill:#3B82F6,color:#fff
|
| 147 |
+
style EDITOR fill:#7C3AED,color:#fff
|
| 148 |
+
style MODEL_LOADER fill:#DC2626,color:#fff
|
| 149 |
+
|
| 150 |
+
linkStyle default stroke:#94A3B8
|
| 151 |
+
```
|
| 152 |
+
|
| 153 |
+
## Critical Dependency Paths
|
| 154 |
+
|
| 155 |
+
```mermaid
|
| 156 |
+
graph LR
|
| 157 |
+
subgraph "Critical Path 1: Text Analysis"
|
| 158 |
+
A1["editor.js"] --> A2["api.js"] --> A3["app.py"] --> A4["model_loader.py"] --> A5["AraSpell"]
|
| 159 |
+
end
|
| 160 |
+
|
| 161 |
+
subgraph "Critical Path 2: Authentication"
|
| 162 |
+
B1["auth-ui.js"] --> B2["auth.js"] --> B3["client.js"] --> B4["Supabase SDK"] --> B5["Supabase"]
|
| 163 |
+
end
|
| 164 |
+
|
| 165 |
+
subgraph "Critical Path 3: Document Sync"
|
| 166 |
+
C1["sync-manager.js"] --> C2["documents-api.js"] --> C3["client.js"] --> C4["Supabase"]
|
| 167 |
+
end
|
| 168 |
+
|
| 169 |
+
style A1 fill:#EF4444,color:#fff
|
| 170 |
+
style B1 fill:#3B82F6,color:#fff
|
| 171 |
+
style C1 fill:#22C55E,color:#fff
|
| 172 |
+
```
|
| 173 |
+
|
| 174 |
+
## Dependency Matrix
|
| 175 |
+
|
| 176 |
+
| Module | Depends On | Depended By |
|
| 177 |
+
|--------|-----------|-------------|
|
| 178 |
+
| `editor.js` | renderer, selection, api, ui | index.html |
|
| 179 |
+
| `renderer.js` | — | editor.js |
|
| 180 |
+
| `auth.js` | client, session | auth-ui.js |
|
| 181 |
+
| `documents-api.js` | client | documents-ui, sync-manager |
|
| 182 |
+
| `sync-manager.js` | sync-queue, sync-resolver, documents-api | editor.js |
|
| 183 |
+
| `app.py` | model_loader, hf_inference, araspell_service | Gunicorn |
|
| 184 |
+
| `model_loader.py` | transformers, torch, huggingface_hub | app.py |
|
| 185 |
+
| `client.js` | config, supabase-js CDN | auth, documents-api, summaries-api, settings-api |
|
| 186 |
+
|
| 187 |
+
## Future Extension Points
|
| 188 |
+
|
| 189 |
+
| Extension | Where to Add | Dependencies Needed |
|
| 190 |
+
|-----------|-------------|-------------------|
|
| 191 |
+
| New NLP model | `model_loader.py` + `/api/` route in `app.py` | transformers |
|
| 192 |
+
| Multi-language | `araspell_rules.py` + new language rules | Language-specific models |
|
| 193 |
+
| Mobile app | New client consuming same `/api/` endpoints | React Native / Flutter |
|
| 194 |
+
| Analytics | New Supabase table + tracking module | — |
|
| 195 |
+
| Real-time collab | Supabase Realtime channel | supabase-js realtime |
|
docs/architecture/11-production-architecture.md
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 11 — Final Production Architecture
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
|
| 5 |
+
This document represents the **FINAL production architecture** of BAYAN, with all five NLP modules fully deployed, integrated, and operational inside a single Docker container on HuggingFace Spaces.
|
| 6 |
+
|
| 7 |
+
## Production Architecture
|
| 8 |
+
|
| 9 |
+
```mermaid
|
| 10 |
+
graph TB
|
| 11 |
+
subgraph "Client (Browser)"
|
| 12 |
+
BROWSER["🌐 Browser<br/>HTML5 + CSS3 + ES6"]
|
| 13 |
+
|
| 14 |
+
subgraph "Frontend SPA"
|
| 15 |
+
LANDING["Landing Page"]
|
| 16 |
+
FEATURES["Features Page"]
|
| 17 |
+
EDITOR_PG["Editor Page"]
|
| 18 |
+
PRICING["Pricing Page"]
|
| 19 |
+
AUTH_GATE["Auth Gate"]
|
| 20 |
+
end
|
| 21 |
+
|
| 22 |
+
subgraph "Editor Engine"
|
| 23 |
+
EDIT["editor.js — State & Analysis"]
|
| 24 |
+
REND["renderer.js — Highlights"]
|
| 25 |
+
SEL["selection.js — Caret"]
|
| 26 |
+
FMT["format.js — Rich Text"]
|
| 27 |
+
end
|
| 28 |
+
|
| 29 |
+
subgraph "Client Services"
|
| 30 |
+
AUTH_SVC["Auth Service"]
|
| 31 |
+
DOC_SVC["Document Manager"]
|
| 32 |
+
SYNC_SVC["Sync Engine"]
|
| 33 |
+
SUMM_SVC["Summary Manager"]
|
| 34 |
+
SET_SVC["Settings Manager"]
|
| 35 |
+
EXPORT_SVC["Export (TXT/DOCX/PDF)"]
|
| 36 |
+
end
|
| 37 |
+
end
|
| 38 |
+
|
| 39 |
+
subgraph "HuggingFace Spaces"
|
| 40 |
+
subgraph "Docker: python:3.12-slim"
|
| 41 |
+
subgraph "Gunicorn (Port 7860, 1 Worker)"
|
| 42 |
+
subgraph "Flask Application"
|
| 43 |
+
STATIC["Static File Server<br/>HTML · CSS · JS · Assets"]
|
| 44 |
+
|
| 45 |
+
subgraph "REST API"
|
| 46 |
+
HEALTH["GET /api/health<br/>Status + Model Readiness"]
|
| 47 |
+
ANALYZE["POST /api/analyze<br/>Full Pipeline: Spell→Grammar→Punct"]
|
| 48 |
+
SPELL_EP["POST /api/spelling<br/>AraSpell Only"]
|
| 49 |
+
GRAMMAR_EP["POST /api/grammar<br/>Grammar Only"]
|
| 50 |
+
PUNCT_EP["POST /api/punctuation<br/>Punctuation Only"]
|
| 51 |
+
SUMM_EP["POST /api/summarize<br/>MBart Summarization"]
|
| 52 |
+
AUTO_EP["POST /api/autocomplete<br/>Word Completion"]
|
| 53 |
+
DEBUG["GET /api/debug/models<br/>Model Status"]
|
| 54 |
+
end
|
| 55 |
+
end
|
| 56 |
+
|
| 57 |
+
subgraph "Model Loader (Singleton)"
|
| 58 |
+
LOADER["ModelLoader<br/>Lazy Init · Error Recovery"]
|
| 59 |
+
end
|
| 60 |
+
|
| 61 |
+
subgraph "NLP Models (Pre-cached in Docker)"
|
| 62 |
+
ARASPELL["🔴 AraSpell<br/>AraBERT Encoder-Decoder<br/>+ last_model.pt<br/>+ ContextualCorrector (MLM)<br/>~220 MB"]
|
| 63 |
+
GRAMMAR["🟡 Bayan Grammar<br/>Rule Engine + ML<br/>~50 MB"]
|
| 64 |
+
PUNCTUATION["🔵 PuncAra-v1<br/>Sequence Labeling<br/>~100 MB"]
|
| 65 |
+
AUTOCOMPLETE["🟢 AutoComplete<br/>Language Model<br/>~100 MB"]
|
| 66 |
+
SUMMARIZATION["🟣 Summarization<br/>MBart (float16)<br/>~600 MB"]
|
| 67 |
+
end
|
| 68 |
+
|
| 69 |
+
subgraph "HF Inference Fallback"
|
| 70 |
+
HF_REMOTE["hf_inference.py<br/>Remote API when RAM limited"]
|
| 71 |
+
end
|
| 72 |
+
end
|
| 73 |
+
end
|
| 74 |
+
end
|
| 75 |
+
|
| 76 |
+
subgraph "External Services"
|
| 77 |
+
SUPABASE["🗄️ Supabase Cloud<br/>PostgreSQL + Auth + RLS"]
|
| 78 |
+
GOOGLE_OAUTH["🔐 Google OAuth 2.0"]
|
| 79 |
+
GITHUB["📦 GitHub<br/>Source Control"]
|
| 80 |
+
GH_ACTIONS["⚙️ GitHub Actions<br/>CI/CD Pipeline"]
|
| 81 |
+
HF_HUB["🤗 HuggingFace Hub<br/>Model Registry"]
|
| 82 |
+
end
|
| 83 |
+
|
| 84 |
+
%% Client to Server
|
| 85 |
+
BROWSER --> STATIC
|
| 86 |
+
EDIT -->|"POST"| ANALYZE
|
| 87 |
+
EDIT -->|"POST"| AUTO_EP
|
| 88 |
+
SUMM_SVC -->|"POST"| SUMM_EP
|
| 89 |
+
|
| 90 |
+
%% API to Models
|
| 91 |
+
ANALYZE --> LOADER
|
| 92 |
+
SPELL_EP --> LOADER
|
| 93 |
+
GRAMMAR_EP --> LOADER
|
| 94 |
+
PUNCT_EP --> LOADER
|
| 95 |
+
SUMM_EP --> LOADER
|
| 96 |
+
AUTO_EP --> LOADER
|
| 97 |
+
|
| 98 |
+
LOADER --> ARASPELL
|
| 99 |
+
LOADER --> GRAMMAR
|
| 100 |
+
LOADER --> PUNCTUATION
|
| 101 |
+
LOADER --> AUTOCOMPLETE
|
| 102 |
+
LOADER --> SUMMARIZATION
|
| 103 |
+
LOADER -.->|"Fallback"| HF_REMOTE
|
| 104 |
+
|
| 105 |
+
%% Client to External
|
| 106 |
+
AUTH_SVC -->|"Supabase SDK"| SUPABASE
|
| 107 |
+
DOC_SVC --> SUPABASE
|
| 108 |
+
SYNC_SVC --> SUPABASE
|
| 109 |
+
SUMM_SVC --> SUPABASE
|
| 110 |
+
SET_SVC --> SUPABASE
|
| 111 |
+
AUTH_SVC -->|"OAuth"| GOOGLE_OAUTH
|
| 112 |
+
|
| 113 |
+
%% Infrastructure
|
| 114 |
+
GITHUB --> GH_ACTIONS
|
| 115 |
+
GH_ACTIONS -->|"Deploy"| HF_HUB
|
| 116 |
+
|
| 117 |
+
%% Styling
|
| 118 |
+
style ARASPELL fill:#EF4444,color:#fff
|
| 119 |
+
style GRAMMAR fill:#F59E0B,color:#000
|
| 120 |
+
style PUNCTUATION fill:#3B82F6,color:#fff
|
| 121 |
+
style AUTOCOMPLETE fill:#10B981,color:#fff
|
| 122 |
+
style SUMMARIZATION fill:#8B5CF6,color:#fff
|
| 123 |
+
style LOADER fill:#7C3AED,color:#fff
|
| 124 |
+
style SUPABASE fill:#3B82F6,color:#fff
|
| 125 |
+
style GOOGLE_OAUTH fill:#DB4437,color:#fff
|
| 126 |
+
```
|
| 127 |
+
|
| 128 |
+
## Resource Budget
|
| 129 |
+
|
| 130 |
+
| Resource | Allocation | Notes |
|
| 131 |
+
|----------|-----------|-------|
|
| 132 |
+
| **RAM** | ~2.5 GB peak | 5 models loaded simultaneously |
|
| 133 |
+
| **Disk** | ~1.07 GB models + ~500 MB system | Pre-cached during Docker build |
|
| 134 |
+
| **CPU** | 2 vCPU (HF Spaces) | Single Gunicorn worker |
|
| 135 |
+
| **Network** | None at runtime | Models pre-downloaded |
|
| 136 |
+
| **Cold Start** | ~30-60 seconds | Model lazy loading |
|
| 137 |
+
|
| 138 |
+
## API Endpoint Summary
|
| 139 |
+
|
| 140 |
+
| Endpoint | Method | Purpose | Model(s) Used |
|
| 141 |
+
|----------|--------|---------|--------------|
|
| 142 |
+
| `/` | GET | Serve SPA + inject Supabase creds | — |
|
| 143 |
+
| `/api/health` | GET | Health check + model status | All (status check) |
|
| 144 |
+
| `/api/analyze` | POST | Full NLP pipeline | AraSpell → Grammar → Punctuation |
|
| 145 |
+
| `/api/spelling` | POST | Spelling correction only | AraSpell |
|
| 146 |
+
| `/api/grammar` | POST | Grammar check only | Grammar |
|
| 147 |
+
| `/api/punctuation` | POST | Punctuation restoration only | PuncAra-v1 |
|
| 148 |
+
| `/api/summarize` | POST | Text summarization | MBart |
|
| 149 |
+
| `/api/autocomplete` | POST | Word completion | AutoComplete |
|
| 150 |
+
| `/api/debug/models` | GET | Model debug info | All |
|
| 151 |
+
|
| 152 |
+
## Production Hardening
|
| 153 |
+
|
| 154 |
+
| Feature | Implementation |
|
| 155 |
+
|---------|---------------|
|
| 156 |
+
| **Error Recovery** | ModelLoader catches load failures; API returns 503 with details |
|
| 157 |
+
| **Timeout** | Gunicorn 120s timeout for long summarizations |
|
| 158 |
+
| **CORS** | Restricted to `/api/*` routes only |
|
| 159 |
+
| **Input Validation** | Max 5000 chars, min 10 chars for summarization |
|
| 160 |
+
| **Startup Errors** | Captured in `_startup_errors` array, exposed via `/api/health` |
|
| 161 |
+
| **HF Fallback** | Auto-routes to HF Inference API when `HF_API_TOKEN` is set |
|
| 162 |
+
| **Model Caching** | All models downloaded during `docker build`, not at runtime |
|
docs/architecture/12-architecture-audit-report.md
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 12 — Architecture Audit Report
|
| 2 |
+
|
| 3 |
+
## System Statistics
|
| 4 |
+
|
| 5 |
+
### Frontend
|
| 6 |
+
|
| 7 |
+
| Category | Count | Files |
|
| 8 |
+
|----------|-------|-------|
|
| 9 |
+
| **Pages** | 5 | Landing, Features, Editor, Pricing, 404 |
|
| 10 |
+
| **Core Editor** | 4 | editor.js, renderer.js, selection.js, format.js |
|
| 11 |
+
| **Auth Modules** | 5 | auth.js, auth-ui.js, session.js, client.js, config.js |
|
| 12 |
+
| **Document Modules** | 3 | documents-api.js, documents-ui.js, documents-state.js |
|
| 13 |
+
| **Summary Modules** | 2 | summaries-api.js, summaries-ui.js |
|
| 14 |
+
| **Settings Modules** | 2 | settings-api.js, settings-sync.js |
|
| 15 |
+
| **Sync Modules** | 3 | sync-manager.js, sync-queue.js, sync-resolver.js |
|
| 16 |
+
| **UI Modules** | 3 | ui.js, theme.js, api.js |
|
| 17 |
+
| **CSS Files** | 2 | main.css, components.css |
|
| 18 |
+
| **Vendor Libraries** | 2 | docx.min.js, jspdf.umd.min.js |
|
| 19 |
+
| **Total Frontend Files** | **28** | |
|
| 20 |
+
|
| 21 |
+
### Backend
|
| 22 |
+
|
| 23 |
+
| Category | Count | Files |
|
| 24 |
+
|----------|-------|-------|
|
| 25 |
+
| **Flask Application** | 1 | app.py (972 lines) |
|
| 26 |
+
| **Model Loader** | 1 | model_loader.py |
|
| 27 |
+
| **HF Inference** | 1 | hf_inference.py |
|
| 28 |
+
| **NLP Services** | 2 | araspell_service.py, araspell_rules.py |
|
| 29 |
+
| **Total Backend Files** | **5** | |
|
| 30 |
+
|
| 31 |
+
### API Endpoints
|
| 32 |
+
|
| 33 |
+
| # | Route | Method |
|
| 34 |
+
|---|-------|--------|
|
| 35 |
+
| 1 | `/` | GET |
|
| 36 |
+
| 2 | `/api/health` | GET |
|
| 37 |
+
| 3 | `/api/analyze` | POST |
|
| 38 |
+
| 4 | `/api/spelling` | POST |
|
| 39 |
+
| 5 | `/api/grammar` | POST |
|
| 40 |
+
| 6 | `/api/punctuation` | POST |
|
| 41 |
+
| 7 | `/api/summarize` | POST |
|
| 42 |
+
| 8 | `/api/autocomplete` | POST |
|
| 43 |
+
| 9 | `/api/debug/models` | GET |
|
| 44 |
+
| **Total** | **9** | |
|
| 45 |
+
|
| 46 |
+
### Database
|
| 47 |
+
|
| 48 |
+
| Table | Columns | RLS |
|
| 49 |
+
|-------|---------|-----|
|
| 50 |
+
| profiles | 6 | ✅ |
|
| 51 |
+
| documents | 6 | ✅ |
|
| 52 |
+
| summaries | 8 | ✅ |
|
| 53 |
+
| settings | 5 | ✅ |
|
| 54 |
+
| **Total** | **4 tables, 25 columns** | |
|
| 55 |
+
|
| 56 |
+
### NLP Models
|
| 57 |
+
|
| 58 |
+
| Model | Architecture | Parameters | Disk |
|
| 59 |
+
|-------|-------------|-----------|------|
|
| 60 |
+
| AraSpell | AraBERT Enc-Dec | ~135M | ~220MB |
|
| 61 |
+
| Grammar | Rules + ML | ~25M | ~50MB |
|
| 62 |
+
| PuncAra-v1 | Seq Labeling | ~50M | ~100MB |
|
| 63 |
+
| AutoComplete | Language Model | ~50M | ~100MB |
|
| 64 |
+
| Summarization | MBart (fp16) | ~610M | ~600MB |
|
| 65 |
+
| **Total** | | **~870M** | **~1.07GB** |
|
| 66 |
+
|
| 67 |
+
---
|
| 68 |
+
|
| 69 |
+
## Completed System Inventory
|
| 70 |
+
|
| 71 |
+
| # | Subsystem | Status | Key Feature |
|
| 72 |
+
|---|-----------|--------|-------------|
|
| 73 |
+
| 1 | ✅ Authentication | Complete | Guest + Google OAuth + Supabase Auth |
|
| 74 |
+
| 2 | ✅ Editor | Complete | Rich text, RTL, undo/redo, formatting |
|
| 75 |
+
| 3 | ✅ Documents | Complete | CRUD, search, sidebar, cloud sync |
|
| 76 |
+
| 4 | ✅ Summaries | Complete | MBart generation, storage, export |
|
| 77 |
+
| 5 | ✅ Settings | Complete | Theme, font, cloud sync |
|
| 78 |
+
| 6 | ✅ Sync Engine | Complete | Offline-first, queue, conflict resolution |
|
| 79 |
+
| 7 | ✅ Offline Engine | Complete | localStorage drafts, recovery |
|
| 80 |
+
| 8 | ✅ Export Engine | Complete | TXT, DOCX (docx.js), PDF (jsPDF) |
|
| 81 |
+
| 9 | ✅ AraSpell | Complete | AraBERT encoder-decoder + alternatives |
|
| 82 |
+
| 10 | ✅ Grammar | Complete | Rule engine + ML corrections |
|
| 83 |
+
| 11 | ✅ Punctuation | Complete | PuncAra-v1 sequence labeling |
|
| 84 |
+
| 12 | ✅ AutoComplete | Complete | Language model word predictions |
|
| 85 |
+
| 13 | ✅ Summarization | Complete | MBart fine-tuned Arabic summarization |
|
| 86 |
+
| 14 | ✅ CI/CD | Complete | GitHub Actions → HF Spaces |
|
| 87 |
+
| 15 | ✅ Health Monitoring | Complete | /api/health + model status |
|
| 88 |
+
|
| 89 |
+
---
|
| 90 |
+
|
| 91 |
+
## Architecture Risks
|
| 92 |
+
|
| 93 |
+
### 🔴 High Risk
|
| 94 |
+
|
| 95 |
+
| Risk | Impact | Mitigation |
|
| 96 |
+
|------|--------|------------|
|
| 97 |
+
| **Model Memory Usage** | 5 models = ~2.5GB RAM; HF free tier = 16GB shared | Float16 models, lazy loading, HF Inference fallback |
|
| 98 |
+
| **Cold Start Latency** | First request after deploy: 30-60s model load | Health check waits; UI shows "جاري التحليل..." |
|
| 99 |
+
| **Single Worker** | One Gunicorn worker = no concurrency | Acceptable for graduation project; scale with more workers later |
|
| 100 |
+
|
| 101 |
+
### 🟡 Medium Risk
|
| 102 |
+
|
| 103 |
+
| Risk | Impact | Mitigation |
|
| 104 |
+
|------|--------|------------|
|
| 105 |
+
| **NLP Inference Latency** | Full pipeline (/api/analyze) can take 2-5s | 1s debounce, "analyzing" indicator, async UI |
|
| 106 |
+
| **Supabase Free Tier** | 500MB database, 50k monthly active users | Sufficient for graduation; upgrade plan available |
|
| 107 |
+
| **No Rate Limiting** | API can be abused | Add Flask-Limiter for production |
|
| 108 |
+
| **No HTTPS Certificate** | HF Spaces handles this | ✅ Handled automatically |
|
| 109 |
+
|
| 110 |
+
### 🟢 Low Risk
|
| 111 |
+
|
| 112 |
+
| Risk | Impact | Mitigation |
|
| 113 |
+
|------|--------|------------|
|
| 114 |
+
| **Browser Compatibility** | contenteditable varies across browsers | Tested on Chrome, Firefox, Safari |
|
| 115 |
+
| **localStorage Limits** | ~5MB per domain | Only stores draft + dismissed words |
|
| 116 |
+
| **CDN Dependency** | Supabase SDK loaded from CDN | Could be vendored if needed |
|
| 117 |
+
|
| 118 |
+
---
|
| 119 |
+
|
| 120 |
+
## Future Expansion Points
|
| 121 |
+
|
| 122 |
+
### Near-Term
|
| 123 |
+
|
| 124 |
+
| Feature | Effort | Dependencies |
|
| 125 |
+
|---------|--------|-------------|
|
| 126 |
+
| **Rate Limiting** | Low | Flask-Limiter package |
|
| 127 |
+
| **Analytics Dashboard** | Medium | New Supabase table + chart library |
|
| 128 |
+
| **Keyboard Shortcuts** | Low | Additional keydown handlers |
|
| 129 |
+
| **Mobile Responsive Polish** | Medium | CSS media queries |
|
| 130 |
+
|
| 131 |
+
### Mid-Term
|
| 132 |
+
|
| 133 |
+
| Feature | Effort | Dependencies |
|
| 134 |
+
|---------|--------|-------------|
|
| 135 |
+
| **Real-time Collaboration** | High | Supabase Realtime + OT/CRDT |
|
| 136 |
+
| **Version History** | Medium | documents_history table |
|
| 137 |
+
| **User Activity Logs** | Low | activity_logs table |
|
| 138 |
+
| **Team/Organization Support** | High | RBAC + team table |
|
| 139 |
+
|
| 140 |
+
### Long-Term
|
| 141 |
+
|
| 142 |
+
| Feature | Effort | Dependencies |
|
| 143 |
+
|---------|--------|-------------|
|
| 144 |
+
| **Multi-Language Support** | Very High | Per-language NLP models |
|
| 145 |
+
| **Mobile App** | High | React Native / Flutter |
|
| 146 |
+
| **Advanced Grammar** | High | Transformer-based grammar model |
|
| 147 |
+
| **Voice Input** | Medium | Web Speech API |
|
| 148 |
+
| **Plugin System** | High | Extension API architecture |
|
| 149 |
+
|
| 150 |
+
---
|
| 151 |
+
|
| 152 |
+
## Architecture Quality Assessment
|
| 153 |
+
|
| 154 |
+
| Criterion | Rating | Notes |
|
| 155 |
+
|-----------|--------|-------|
|
| 156 |
+
| **Separation of Concerns** | ⭐⭐⭐⭐ | Clear layers: UI → API → NLP → DB |
|
| 157 |
+
| **Modularity** | ⭐⭐⭐⭐ | Each module has single responsibility |
|
| 158 |
+
| **Testability** | ⭐⭐⭐ | API endpoints testable; frontend needs E2E |
|
| 159 |
+
| **Scalability** | ⭐⭐⭐ | Vertical scaling possible; horizontal needs work |
|
| 160 |
+
| **Security** | ⭐⭐⭐⭐ | RLS, CORS, input validation, OAuth |
|
| 161 |
+
| **Maintainability** | ⭐⭐⭐⭐ | Clean file structure, documented APIs |
|
| 162 |
+
| **Performance** | ⭐⭐⭐ | Debouncing, lazy loading, float16 |
|
| 163 |
+
| **Reliability** | ⭐⭐⭐⭐ | Offline support, error recovery, fallbacks |
|
| 164 |
+
| **Documentation** | ⭐⭐⭐⭐⭐ | 12 architecture documents with diagrams |
|
| 165 |
+
|
| 166 |
+
---
|
| 167 |
+
|
| 168 |
+
## File Tree (Production)
|
| 169 |
+
|
| 170 |
+
```
|
| 171 |
+
BAYAN/
|
| 172 |
+
├── .github/workflows/deploy.yml # CI/CD pipeline
|
| 173 |
+
├── Dockerfile # Container definition
|
| 174 |
+
├── requirements.txt # Python dependencies
|
| 175 |
+
├── supabase/migrations/001_profiles.sql # Database schema
|
| 176 |
+
├── docs/architecture/ # ← YOU ARE HERE
|
| 177 |
+
│ ├── 01-system-overview.md
|
| 178 |
+
│ ├── 02-class-diagram.md
|
| 179 |
+
│ ├── 03-component-diagram.md
|
| 180 |
+
│ ├── 04-sequence-diagrams.md
|
| 181 |
+
│ ├── 05-dataflow-diagram.md
|
| 182 |
+
│ ├── 06-deployment-diagram.md
|
| 183 |
+
│ ├── 07-database-schema.md
|
| 184 |
+
│ ├── 08-nlp-pipeline-diagram.md
|
| 185 |
+
│ ├── 09-sync-engine-diagram.md
|
| 186 |
+
│ ├── 10-project-dependency-map.md
|
| 187 |
+
│ ├── 11-production-architecture.md
|
| 188 |
+
│ └── 12-architecture-audit-report.md
|
| 189 |
+
└── src/
|
| 190 |
+
├── app.py # Flask backend (972 lines)
|
| 191 |
+
├── model_loader.py # Model management
|
| 192 |
+
├── hf_inference.py # Remote HF API
|
| 193 |
+
├── index.html # SPA (1289 lines)
|
| 194 |
+
├── nlp/
|
| 195 |
+
│ ├── spelling/
|
| 196 |
+
│ │ ├── araspell_service.py
|
| 197 |
+
│ │ └── araspell_rules.py
|
| 198 |
+
│ └── grammar/
|
| 199 |
+
│ └── grammar_service.py
|
| 200 |
+
├── css/
|
| 201 |
+
│ ├── main.css
|
| 202 |
+
│ └── components.css
|
| 203 |
+
└── js/
|
| 204 |
+
├── editor.js # Core editor (611 lines)
|
| 205 |
+
├── renderer.js # Error highlighting
|
| 206 |
+
├── selection.js # Caret management
|
| 207 |
+
├── format.js # Rich text formatting
|
| 208 |
+
├── ui.js # UI utilities
|
| 209 |
+
├── api.js # HTTP client
|
| 210 |
+
├── theme.js # Dark/light mode
|
| 211 |
+
├── auth/
|
| 212 |
+
│ ├── auth.js
|
| 213 |
+
│ ├── auth-ui.js
|
| 214 |
+
│ ├── session.js
|
| 215 |
+
│ ├── client.js
|
| 216 |
+
│ └── config.js
|
| 217 |
+
├── documents-cloud/
|
| 218 |
+
│ ├── documents-api.js
|
| 219 |
+
│ ├── documents-ui.js
|
| 220 |
+
│ └── documents-state.js
|
| 221 |
+
├── summaries/
|
| 222 |
+
│ ├── summaries-api.js
|
| 223 |
+
│ └── summaries-ui.js
|
| 224 |
+
├── settings-sync/
|
| 225 |
+
│ ├── settings-api.js
|
| 226 |
+
│ └── settings-sync.js
|
| 227 |
+
├── sync/
|
| 228 |
+
│ ├── sync-manager.js
|
| 229 |
+
│ ├── sync-queue.js
|
| 230 |
+
│ └── sync-resolver.js
|
| 231 |
+
└── vendor/
|
| 232 |
+
├── docx.min.js
|
| 233 |
+
└── jspdf.umd.min.js
|
| 234 |
+
```
|
| 235 |
+
|
| 236 |
+
---
|
| 237 |
+
|
| 238 |
+
*Generated for BAYAN Graduation Project — Final Production Architecture Documentation*
|
| 239 |
+
*Date: June 2026*
|