---
license: mit
datasets:
- agentlans/openbmb-UltraX-Preview-line-classification
language:
- en
pipeline_tag: text-classification
library_name: fasttext
tags:
- fasttext
- text-classification
- data-cleaning
- quality-filtering
- ultrax
---
# Line Classifier

A lightweight model built with [FastText](https://fasttext.cc/) designed to evaluate individual lines of text and classify them as **keep**, **delete**, or **edit** to improve overall corpus quality.

> [!NOTE]
> This project is inspired by and based on an analysis of the [UltraX project](https://huggingface.co/datasets/openbmb/UltraX-Preview). It is not officially affiliated with it.

## 🏋️ Training & Methodology

1. The model (`line_classifier.bin`) was trained using FastText's supervised learning with **autotune** enabled, utilizing the train and validation splits from [`agentlans/openbmb-UltraX-Preview-line-classification`](https://huggingface.co/datasets/agentlans/openbmb-UltraX-Preview-line-classification). 
2. After training, the model was compressed using FastText's built-in quantization (`.ftz` format) to significantly reduce file size and memory footprint while maintaining robust classification performance.
3. Evaluated on the independent test split of `agentlans/openbmb-UltraX-Preview-line-classification`, achieving the following overall metrics:

| Metric | Score | Sample Size (N) |
| :--- | :---: | :---: |
| **Precision (@1)** | 0.88 | 526238 |
| **Recall (@1)** | 0.88 | 526238 |

## 🚀 Quick Start

1. Install FastText

```bash
pip install fasttext
```

2. Download the [`line_classifier.bin`](https://huggingface.co/agentlans/fasttext-line-classifier/resolve/main/line_classifier.bin) or [`line_classifier.ftz`](https://huggingface.co/agentlans/fasttext-line-classifier/resolve/main/line_classifier.ftz) file from this repository, then load and run predictions using Python:

```python
import fasttext

# Load the model
model = fasttext.load_model("line_classifier.ftz")

# Test the model on sample text lines
texts = [
    "You are the 100th visitor today! Claim your prize NOW!",
    "This lecture series will examine the role of sweepstakes on consumer behaviour."
]

labels, probabilities = model.predict(texts)
print(labels)
print(probabilities)

```

## 📊 Example Predictions

| Line of Text | Prediction | Confidence |
| --- | --- | --- |
| `Type: Whitepaper Poster` | `delete` | 0.4536 |
| `We now have FIVE dogs! I never thought I would have five dogs, but at least two are small dogs (the Boston Terriers — Guinness and Rollie) and two are calm senior dogs (Dusty and Tansy). [...]` | `keep` | 0.9354 |
| `The Gifu Prefecture on the main island of Honshu and the state of Baden-Württemberg have long nurtured their political and economic ties...` | `keep` | 0.9527 |
| `## LIVE TICKET FOR 1` | `edit` | 0.4407 |
| `Nclex Categories Breakdown` | `delete` | 0.8325 |

## ⚠️ Limitations & Considerations

* **Lack of Context:** The model operates strictly one line at a time. It lacks broader contextual awareness, which can occasionally make it difficult to determine the true importance of a line.
* **Repetitive Content:** Standalone line evaluation means duplicate or repetitive lines across a document may not always be successfully flagged or filtered out.
* **Structured Content:** The model may struggle with or mangle heavily formatted text layouts, such as Markdown tables, ASCII art, or structural diagrams.

## 📄 License

Distributed under the **MIT License**. See the [LICENSE](https://www.google.com/search?q=LICENSE) file for details.