Business Issue Allocation Classifier
A text classifier that maps a natural language business problem description to the most likely data engineering solution category.
💡 How to state a business issue
Write 1-2 sentences the model can classify reliably:
- Describe the data situation - what systems and data you have, and why the need is hard to meet. Don't just express frustration ("I have no idea what's happening").
- State the desired outcome - the concrete result or constraint you want ("one consolidated monthly report", "alerts within milliseconds").
- Describe the need, not the solution - don't name or negate a category; "we don't have a data warehouse" leaks the answer and confuses the model. Let it infer the label.
- Keep it to one issue - one problem per description, not several bundled together.
❌ Weak: "I have no idea what's happening with our finances. I don't see profits per category. We don't have a data warehouse."
✅ Strong: "Our finance data is spread across several systems and we can't get profit broken down by product category. We need one consolidated place with monthly profit-by-category figures for management reporting."
Model Details
- Classifier: SVM (Support Vector Machine)
- Embedding model:
sentence-transformers/all-mpnet-base-v2 - Classes: 9 (stream_processing, etl_pipeline, data_warehouse, data_lake, api_integration, ml_feature_store, data_caching, data_governance, data_quality)
- Accuracy: 88.2%
- Macro F1: 88.4%
How to Use
Clone the full project from GitHub and run:
from src.inference import Predictor
predictor = Predictor()
result = predictor.predict("We need to detect fraud before transactions are approved.")
print(result["predicted_label"])
Label distribution in dataset
The dataset contains 465 labeled examples across 9 solution categories (label_it).
The classes are fairly balanced, ranging from 39 to 63 examples per category.
| Label | Count | Share |
|---|---|---|
data_warehouse |
63 | 13.5% |
api_integration |
61 | 13.1% |
data_governance |
60 | 12.9% |
stream_processing |
51 | 11.0% |
etl_pipeline |
48 | 10.3% |
data_lake |
48 | 10.3% |
ml_feature_store |
48 | 10.3% |
data_caching |
47 | 10.1% |
data_quality |
39 | 8.4% |
NOTES FROM TESTING
The model favoured label "data_governance" in these cases: I have no idea, what is happening with our finances. I don't see anywhere profits per category. We don't have data warehouse.
Dataset
dianamikova/business-issue-allocation