Update README.md
Browse files
README.md
CHANGED
|
@@ -8,7 +8,55 @@ pipeline_tag: text-classification
|
|
| 8 |
library_name: fasttext
|
| 9 |
tags:
|
| 10 |
- fasttext
|
| 11 |
-
-
|
| 12 |
-
-
|
| 13 |
-
-
|
| 14 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
library_name: fasttext
|
| 9 |
tags:
|
| 10 |
- fasttext
|
| 11 |
+
- text-classification
|
| 12 |
+
- data-cleaning
|
| 13 |
+
- quality-filtering
|
| 14 |
+
- ultrax
|
| 15 |
+
---
|
| 16 |
+
# UltraX Line Classifier
|
| 17 |
+
|
| 18 |
+
A lightweight model built with [FastText](https://fasttext.cc/) designed to evaluate individual lines of text and classify them as **keep**, **delete**, or **edit** to improve overall corpus quality.
|
| 19 |
+
|
| 20 |
+
**Disclaimer:** This project is inspired by and based on an analysis of the [UltraX project](https://huggingface.co/datasets/openbmb/UltraX-Preview) (it is not officially affiliated with it).
|
| 21 |
+
|
| 22 |
+
## 🚀 Quick Start
|
| 23 |
+
|
| 24 |
+
First, download the `line_classifier.ftz` file from this repository, then load and run predictions using Python:
|
| 25 |
+
|
| 26 |
+
```python
|
| 27 |
+
import fasttext
|
| 28 |
+
|
| 29 |
+
# Load the model
|
| 30 |
+
model = fasttext.load_model("line_classifier.ftz")
|
| 31 |
+
|
| 32 |
+
# Test the model on sample text lines
|
| 33 |
+
texts = [
|
| 34 |
+
"You are the 100th visitor today! Claim your prize NOW!",
|
| 35 |
+
"This lecture series will examine the role of sweepstakes on consumer behaviour."
|
| 36 |
+
]
|
| 37 |
+
|
| 38 |
+
labels, probabilities = model.predict(texts)
|
| 39 |
+
print(labels)
|
| 40 |
+
print(probabilities)
|
| 41 |
+
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
## 📊 Example Predictions
|
| 45 |
+
|
| 46 |
+
| Line of Text | Prediction | Confidence |
|
| 47 |
+
| --- | --- | --- |
|
| 48 |
+
| `Type: Whitepaper Poster` | `delete` | 0.4536 |
|
| 49 |
+
| `We now have FIVE dogs! I never thought I would have five dogs, but at least two are small dogs (the Boston Terriers — Guinness and Rollie) and two are calm senior dogs (Dusty and Tansy). [...]` | `keep` | 0.9354 |
|
| 50 |
+
| `The Gifu Prefecture on the main island of Honshu and the state of Baden-Württemberg have long nurtured their political and economic ties...` | `keep` | 0.9527 |
|
| 51 |
+
| `## LIVE TICKET FOR 1` | `edit` | 0.4407 |
|
| 52 |
+
| `Nclex Categories Breakdown` | `delete` | 0.8325 |
|
| 53 |
+
|
| 54 |
+
## ⚠️ Limitations & Considerations
|
| 55 |
+
|
| 56 |
+
* **Lack of Context:** The model operates strictly one line at a time. It lacks broader contextual awareness, which can occasionally make it difficult to determine the true importance of a line.
|
| 57 |
+
* **Repetitive Content:** Standalone line evaluation means duplicate or repetitive lines across a document may not always be successfully flagged or filtered out.
|
| 58 |
+
* **Structured Content:** The model may struggle with or mangle heavily formatted text layouts, such as Markdown tables, ASCII art, or structural diagrams.
|
| 59 |
+
|
| 60 |
+
## 📄 License
|
| 61 |
+
|
| 62 |
+
Distributed under the **MIT License**. See the [LICENSE](https://www.google.com/search?q=LICENSE) file for details.
|