| --- |
| license: mit |
| datasets: |
| - agentlans/openbmb-UltraX-Preview-line-classification |
| language: |
| - en |
| pipeline_tag: text-classification |
| library_name: fasttext |
| tags: |
| - fasttext |
| - text-classification |
| - data-cleaning |
| - quality-filtering |
| - ultrax |
| --- |
| # Line Classifier |
|
|
| A lightweight model built with [FastText](https://fasttext.cc/) designed to evaluate individual lines of text and classify them as **keep**, **delete**, or **edit** to improve overall corpus quality. |
|
|
| > [!NOTE] |
| > This project is inspired by and based on an analysis of the [UltraX project](https://huggingface.co/datasets/openbmb/UltraX-Preview). It is not officially affiliated with it. |
|
|
| ## 🏋️ Training & Methodology |
|
|
| 1. The model (`line_classifier.bin`) was trained using FastText's supervised learning with **autotune** enabled, utilizing the train and validation splits from [`agentlans/openbmb-UltraX-Preview-line-classification`](https://huggingface.co/datasets/agentlans/openbmb-UltraX-Preview-line-classification). |
| 2. After training, the model was compressed using FastText's built-in quantization (`.ftz` format) to significantly reduce file size and memory footprint while maintaining robust classification performance. |
| 3. Evaluated on the independent test split of `agentlans/openbmb-UltraX-Preview-line-classification`, achieving the following overall metrics: |
|
|
| | Metric | Score | Sample Size (N) | |
| | :--- | :---: | :---: | |
| | **Precision (@1)** | 0.88 | 526238 | |
| | **Recall (@1)** | 0.88 | 526238 | |
|
|
| ## 🚀 Quick Start |
|
|
| 1. Install FastText |
|
|
| ```bash |
| pip install fasttext |
| ``` |
|
|
| 2. Download the [`line_classifier.bin`](https://huggingface.co/agentlans/fasttext-line-classifier/resolve/main/line_classifier.bin) or [`line_classifier.ftz`](https://huggingface.co/agentlans/fasttext-line-classifier/resolve/main/line_classifier.ftz) file from this repository, then load and run predictions using Python: |
|
|
| ```python |
| import fasttext |
| |
| # Load the model |
| model = fasttext.load_model("line_classifier.ftz") |
| |
| # Test the model on sample text lines |
| texts = [ |
| "You are the 100th visitor today! Claim your prize NOW!", |
| "This lecture series will examine the role of sweepstakes on consumer behaviour." |
| ] |
| |
| labels, probabilities = model.predict(texts) |
| print(labels) |
| print(probabilities) |
| |
| ``` |
|
|
| ## 📊 Example Predictions |
|
|
| | Line of Text | Prediction | Confidence | |
| | --- | --- | --- | |
| | `Type: Whitepaper Poster` | `delete` | 0.4536 | |
| | `We now have FIVE dogs! I never thought I would have five dogs, but at least two are small dogs (the Boston Terriers — Guinness and Rollie) and two are calm senior dogs (Dusty and Tansy). [...]` | `keep` | 0.9354 | |
| | `The Gifu Prefecture on the main island of Honshu and the state of Baden-Württemberg have long nurtured their political and economic ties...` | `keep` | 0.9527 | |
| | `## LIVE TICKET FOR 1` | `edit` | 0.4407 | |
| | `Nclex Categories Breakdown` | `delete` | 0.8325 | |
|
|
| ## ⚠️ Limitations & Considerations |
|
|
| * **Lack of Context:** The model operates strictly one line at a time. It lacks broader contextual awareness, which can occasionally make it difficult to determine the true importance of a line. |
| * **Repetitive Content:** Standalone line evaluation means duplicate or repetitive lines across a document may not always be successfully flagged or filtered out. |
| * **Structured Content:** The model may struggle with or mangle heavily formatted text layouts, such as Markdown tables, ASCII art, or structural diagrams. |
|
|
| ## 📄 License |
|
|
| Distributed under the **MIT License**. See the [LICENSE](https://www.google.com/search?q=LICENSE) file for details. |
|
|
