File size: 3,572 Bytes
95a2073 880a827 ff229c8 c43dfe3 ff229c8 c6187f6 3691566 ff229c8 c081b44 3691566 c081b44 ff229c8 3691566 ff229c8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: mit
datasets:
- agentlans/openbmb-UltraX-Preview-line-classification
language:
- en
pipeline_tag: text-classification
library_name: fasttext
tags:
- fasttext
- text-classification
- data-cleaning
- quality-filtering
- ultrax
---
# Line Classifier
A lightweight model built with [FastText](https://fasttext.cc/) designed to evaluate individual lines of text and classify them as **keep**, **delete**, or **edit** to improve overall corpus quality.
> [!NOTE]
> This project is inspired by and based on an analysis of the [UltraX project](https://huggingface.co/datasets/openbmb/UltraX-Preview). It is not officially affiliated with it.
## 🏋️ Training & Methodology
1. The model (`line_classifier.bin`) was trained using FastText's supervised learning with **autotune** enabled, utilizing the train and validation splits from [`agentlans/openbmb-UltraX-Preview-line-classification`](https://huggingface.co/datasets/agentlans/openbmb-UltraX-Preview-line-classification).
2. After training, the model was compressed using FastText's built-in quantization (`.ftz` format) to significantly reduce file size and memory footprint while maintaining robust classification performance.
3. Evaluated on the independent test split of `agentlans/openbmb-UltraX-Preview-line-classification`, achieving the following overall metrics:
| Metric | Score | Sample Size (N) |
| :--- | :---: | :---: |
| **Precision (@1)** | 0.88 | 526238 |
| **Recall (@1)** | 0.88 | 526238 |
## 🚀 Quick Start
1. Install FastText
```bash
pip install fasttext
```
2. Download the [`line_classifier.bin`](https://huggingface.co/agentlans/fasttext-line-classifier/resolve/main/line_classifier.bin) or [`line_classifier.ftz`](https://huggingface.co/agentlans/fasttext-line-classifier/resolve/main/line_classifier.ftz) file from this repository, then load and run predictions using Python:
```python
import fasttext
# Load the model
model = fasttext.load_model("line_classifier.ftz")
# Test the model on sample text lines
texts = [
"You are the 100th visitor today! Claim your prize NOW!",
"This lecture series will examine the role of sweepstakes on consumer behaviour."
]
labels, probabilities = model.predict(texts)
print(labels)
print(probabilities)
```
## 📊 Example Predictions
| Line of Text | Prediction | Confidence |
| --- | --- | --- |
| `Type: Whitepaper Poster` | `delete` | 0.4536 |
| `We now have FIVE dogs! I never thought I would have five dogs, but at least two are small dogs (the Boston Terriers — Guinness and Rollie) and two are calm senior dogs (Dusty and Tansy). [...]` | `keep` | 0.9354 |
| `The Gifu Prefecture on the main island of Honshu and the state of Baden-Württemberg have long nurtured their political and economic ties...` | `keep` | 0.9527 |
| `## LIVE TICKET FOR 1` | `edit` | 0.4407 |
| `Nclex Categories Breakdown` | `delete` | 0.8325 |
## ⚠️ Limitations & Considerations
* **Lack of Context:** The model operates strictly one line at a time. It lacks broader contextual awareness, which can occasionally make it difficult to determine the true importance of a line.
* **Repetitive Content:** Standalone line evaluation means duplicate or repetitive lines across a document may not always be successfully flagged or filtered out.
* **Structured Content:** The model may struggle with or mangle heavily formatted text layouts, such as Markdown tables, ASCII art, or structural diagrams.
## 📄 License
Distributed under the **MIT License**. See the [LICENSE](https://www.google.com/search?q=LICENSE) file for details.
|