Create index.html

index.html

@@ -1,19 +1,150 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
-</html>
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>IfGPT DATASET Quality Components</title>
+<style>
+  body {
+    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif;
+    max-width: 900px;
+    margin: 2em auto;
+    padding: 0 1.5em;
+    line-height: 1.6;
+    color: #24292f;
+    background-color: #ffffff;
+  }
+  h1 {
+    border-bottom: 2px solid #d0d7de;
+    padding-bottom: 0.3em;
+    font-size: 2em;
+  }
+  h2 {
+    border-bottom: 1px solid #d0d7de;
+    padding-bottom: 0.3em;
+    margin-top: 1.8em;
+    font-size: 1.5em;
+  }
+  code {
+    background-color: #f6f8fa;
+    padding: 0.2em 0.4em;
+    border-radius: 4px;
+    font-family: "SF Mono", Menlo, Consolas, monospace;
+    font-size: 0.9em;
+  }
+  pre {
+    background-color: #f6f8fa;
+    padding: 1em;
+    border-radius: 6px;
+    overflow-x: auto;
+    font-family: "SF Mono", Menlo, Consolas, monospace;
+    font-size: 0.9em;
+  }
+  pre code {
+    background: none;
+    padding: 0;
+  }
+  ul {
+    padding-left: 1.5em;
+  }
+  li {
+    margin-bottom: 0.6em;
+  }
+  hr {
+    border: none;
+    border-top: 1px solid #d0d7de;
+    margin: 2em 0;
+  }
+  a {
+    color: #0969da;
+    text-decoration: none;
+  }
+  a:hover {
+    text-decoration: underline;
+  }
+  .footer {
+    font-size: 0.9em;
+    color: #57606a;
+  }
+</style>
+</head>
+<body>
+
+<h1>IfGPT DATASET Quality Components</h1>
+
+<h2>Objectives of the project IfGPT</h2>
+<p>
+  The <strong>IfGPT Quality Pipeline</strong> is developed within the project
+  <strong>IfGPT: Infrastructure for Fine-tuning Pre-trained Large Language Models</strong>,
+  which aims to establish a freely accessible infrastructure for the selection and
+  pre-processing of large datasets for Bulgarian as well as tailored data for particular
+  industries and fine-tuning suitable freely available large language models for specific purposes.
+</p>
+
+<h2>IfGPT Dataset Quality Pipeline</h2>
+<p>
+  Modular Java pipeline to process and add new text documents to the IfGPT Dataset,
+  which includes cleaning, deduplication and quality evaluation of Bulgarian texts.
+</p>
+<p>The pipeline includes:</p>
+<ul>
+  <li>
+    <strong>Source-specific extraction</strong> — metadata and plain text extracted from
+    heterogeneous corpora (MARCELL, CURLICAT, BulNC, Wikipedia), each handled by a
+    dedicated class that implements the shared <code>SourceProcessor</code> interface and
+    extends <code>BaseSourceProcessor</code>.
+  </li>
+  <li>
+    <strong>Sentence splitting</strong> — <code>BulgarianSentenceSplitter</code> wraps the
+    Apache OpenNLP Bulgarian UD sentence-detection model, splitting every document into a
+    sentence-per-line sidecar file used by all downstream stages.
+  </li>
+  <li>
+    <strong>Boilerplate cleaning</strong> — <code>FileCleanProcessor</code> learns
+    site-specific boilerplate from a sample directory (lines appearing in ≥ 50 % of files)
+    and removes them alongside hardcoded patterns for HTML tags, navigation menus, URLs,
+    cookie banners, and more.
+  </li>
+  <li>
+    <strong>MinHash / LSH deduplication</strong> — <code>DeduplicationProcessor</code>
+    builds a MinHash signature index over the full existing corpus and detects
+    near-duplicate sentences in the new batch (Jaccard ≥ 0.90), writing a ranked TSV
+    report and optionally removing duplicates.
+  </li>
+  <li>
+    <strong>Per-sentence PII scoring</strong> — <code>PIIDetector</code> runs every
+    sentence through the Phileas engine (names, emails, phone numbers, IBANs, IP
+    addresses, etc.) and stores the proportion of flagged tokens as the
+    <code>PersonallyIdentifiableInformation</code> vector in metadata.
+  </li>
+  <li>
+    <strong>Per-sentence bias scoring</strong> — <code>BiasAnalyser</code> matches tokens
+    against <code>BiasLexicon</code> (3 787-entry Bulgarian Bias Dictionary v4) to detect
+    signal–evaluator pairs across five categories (gender, race/ethnicity, religion,
+    disability, appearance), storing the per-sentence coverage ratio as the
+    <code>BiasedInformation</code> vector in metadata.
+  </li>
+</ul>
+
+<p>
+  The full schema is enforced by <code>DocumentMetadata</code> (15 mandatory + 8 optional
+  fields) and the complete flow is managed by <code>IfGPTPipeline</code>, with
+  <code>IfGPTDatasetProcessor</code> as the main entry point.
+</p>
+
+<pre><code>source processors → sentence split → clean → deduplication → PII → bias → counts → final structuring</code></pre>
+
+<h2>License</h2>
+<p>Creative Commons Attribution 4.0 International (CC-BY-4.0)</p>
+
+<hr>
+
+<p class="footer">
+  This work is part of the project <strong>Infrastructure for Fine-tuning Pre-trained
+  Large Language Models</strong>, Grant Agreement No. ПВУ – 55 from 12.12.2024
+  /BG-RRP-2.017-0030-C01/.<br>
+  <a href="https://ifgpt.dcl.bas.bg/en/">https://ifgpt.dcl.bas.bg/en/</a>
+</p>
+
+</body>
+</html>