Text Classification
Scikit-learn
Portuguese
intent-classification
lead-scoring
tf-idf
logistic-regression
portuguese
Instructions to use ranzouu/leadforge-intent-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Scikit-learn
How to use ranzouu/leadforge-intent-classifier with Scikit-learn:
from huggingface_hub import hf_hub_download import joblib model = joblib.load( hf_hub_download("ranzouu/leadforge-intent-classifier", "sklearn_model.joblib") ) # only load pickle files from sources you trust # read more about it here https://skops.readthedocs.io/en/stable/persistence.html - Notebooks
- Google Colab
- Kaggle
Upload folder using huggingface_hub
Browse files- LICENSE +21 -0
- README.md +80 -0
- data/sample_leads.json +62 -0
- experiment.py +340 -0
- model.pkl +3 -0
- runs/class_report.csv +4 -0
- runs/cv_folds.csv +6 -0
- runs/summary.csv +2 -0
- vectorizer.pkl +3 -0
LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Rayson Severiano
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
README.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: sklearn
|
| 3 |
+
language: pt
|
| 4 |
+
license: mit
|
| 5 |
+
pipeline_tag: text-classification
|
| 6 |
+
tags:
|
| 7 |
+
- text-classification
|
| 8 |
+
- intent-classification
|
| 9 |
+
- lead-scoring
|
| 10 |
+
- tf-idf
|
| 11 |
+
- logistic-regression
|
| 12 |
+
- portuguese
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# LeadForge Intent Classifier
|
| 16 |
+
|
| 17 |
+
Classificador de **intenção de prospecção** (`hot` / `warm` / `cold`) a partir do texto do lead, em português. Pipeline TF-IDF (word n-grams + char_wb n-grams) + Logistic Regression balanceada — sem API paga, sem GPU, determinístico (`random_state=42`).
|
| 18 |
+
|
| 19 |
+
Faz parte do pipeline do [LeadForge](https://github.com/ranzouu/leadforge). Código completo, CLI e testes: [github.com/ranzouu/leadforge-intent-classifier](https://github.com/ranzouu/leadforge-intent-classifier).
|
| 20 |
+
|
| 21 |
+
## Métricas — avaliação honesta (out-of-fold)
|
| 22 |
+
|
| 23 |
+
**StratifiedKFold (5 folds) com tuning de `C` aninhado dentro do fold de treino.** Os números são pooled out-of-fold: toda amostra foi classificada por um modelo que nunca a viu no treino.
|
| 24 |
+
|
| 25 |
+
| Métrica (média ± desvio entre folds) | Valor |
|
| 26 |
+
|---|---|
|
| 27 |
+
| Accuracy | 0.633 ± 0.192 |
|
| 28 |
+
| Precision (weighted) | 0.634 ± 0.197 |
|
| 29 |
+
| Recall (weighted) | 0.633 ± 0.192 |
|
| 30 |
+
| F1 (weighted) | 0.630 ± 0.189 |
|
| 31 |
+
| F1 (macro) | 0.633 |
|
| 32 |
+
|
| 33 |
+
| Classe | Precision | Recall | F1 | Suporte |
|
| 34 |
+
|---|---|---|---|---|
|
| 35 |
+
| cold | 0.700 | 0.700 | 0.700 | 20 |
|
| 36 |
+
| hot | 0.750 | 0.750 | 0.750 | 20 |
|
| 37 |
+
| warm | 0.450 | 0.450 | 0.450 | 20 |
|
| 38 |
+
|
| 39 |
+
> Nota de integridade: uma versão anterior deste projeto publicava "accuracy 1.0" medida no mesmo conjunto usado no treino (avaliação in-sample — inflação). Esse número foi removido e substituído pela validação cruzada acima. Detalhes por fold em `runs/cv_folds.csv` e por classe em `runs/class_report.csv`.
|
| 40 |
+
|
| 41 |
+
## Uso
|
| 42 |
+
|
| 43 |
+
```python
|
| 44 |
+
import joblib
|
| 45 |
+
|
| 46 |
+
vectorizer = joblib.load("vectorizer.pkl")
|
| 47 |
+
clf = joblib.load("model.pkl")
|
| 48 |
+
|
| 49 |
+
frases = [
|
| 50 |
+
"Quero fechar um plano hoje para minha barbearia", # hot
|
| 51 |
+
"Pode me explicar melhor como funciona?", # warm
|
| 52 |
+
"Só estou olhando por curiosidade", # cold
|
| 53 |
+
]
|
| 54 |
+
print(clf.predict(vectorizer.transform([f.lower() for f in frases])))
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
## Limitações (lé com atenção)
|
| 58 |
+
|
| 59 |
+
- **Dataset de demonstração**: 60 amostras (20 por classe) escritas para exercitar o pipeline — não são leads reais de produção. Em produção, treinar com dados reais e revalidar antes de confiar.
|
| 60 |
+
- **`warm` é a fronteira difícil** (F1 0.45): separar "interesse" de "só curiosidade" é tênue até em leitura humana.
|
| 61 |
+
- Modelo linha de base (TF-IDF + LR); com dados reais e volume maior, comparar com alternativas (LinearSVC, gradient boosting, LLM zero-shot) antes de escolher.
|
| 62 |
+
|
| 63 |
+
## Reproduzir
|
| 64 |
+
|
| 65 |
+
```bash
|
| 66 |
+
pip install -e . # do repositório
|
| 67 |
+
python -m leadforge_intent_classifier
|
| 68 |
+
# runs/summary.csv, runs/cv_folds.csv, runs/class_report.csv, runs/latest.csv
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
## Arquivos
|
| 72 |
+
|
| 73 |
+
| Arquivo | Conteúdo |
|
| 74 |
+
|---|---|
|
| 75 |
+
| `model.pkl` | Logistic Regression final (treinada nos 60 samples, `C=4.0`) |
|
| 76 |
+
| `vectorizer.pkl` | FeatureUnion word(1,2) + char_wb(2,4), `sublinear_tf` |
|
| 77 |
+
| `experiment.py` | código-fonte do pipeline (treino, nested CV, exportação) |
|
| 78 |
+
| `data/sample_leads.json` | dataset de demo (60 amostras) |
|
| 79 |
+
| `runs/*.csv` | métricas por fold, por classe e predições out-of-fold |
|
| 80 |
+
| `LICENSE` | MIT |
|
data/sample_leads.json
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{"id": 1, "text": "Quero contratar um plano de musculação para minha academia", "label": "hot"},
|
| 3 |
+
{"id": 2, "text": "Preciso de um site para meu petshop", "label": "hot"},
|
| 4 |
+
{"id": 3, "text": "Quero automatizar o cardápio do meu restaurante", "label": "hot"},
|
| 5 |
+
{"id": 4, "text": "Estou procurando um sistema para minha loja", "label": "warm"},
|
| 6 |
+
{"id": 5, "text": "Quero melhorar a organização da minha academia", "label": "warm"},
|
| 7 |
+
{"id": 6, "text": "Meu petshop precisa de um sistema de agenda", "label": "warm"},
|
| 8 |
+
{"id": 7, "text": "Quero entender mais sobre automação", "label": "cold"},
|
| 9 |
+
{"id": 8, "text": "Estou apenas pesquisando opções", "label": "cold"},
|
| 10 |
+
{"id": 9, "text": "Quero um aplicativo para meu negócio", "label": "hot"},
|
| 11 |
+
{"id": 10, "text": "Preciso de ajuda com marketing digital", "label": "warm"},
|
| 12 |
+
{"id": 11, "text": "Só queria ver como funciona", "label": "cold"},
|
| 13 |
+
{"id": 12, "text": "Quero contratar um personal para minha academia", "label": "hot"},
|
| 14 |
+
{"id": 13, "text": "Estou começando um petshop e preciso de ajuda", "label": "warm"},
|
| 15 |
+
{"id": 14, "text": "Quero criar um site institucional", "label": "hot"},
|
| 16 |
+
{"id": 15, "text": "Ainda não sei se preciso de um sistema", "label": "cold"},
|
| 17 |
+
{"id": 16, "text": "Quero fechar um plano hoje para minha barbearia", "label": "hot"},
|
| 18 |
+
{"id": 17, "text": "Preciso contratar um sistema pro meu salão essa semana", "label": "hot"},
|
| 19 |
+
{"id": 18, "text": "Quero um site pra minha loja de roupas, pode mandar proposta", "label": "hot"},
|
| 20 |
+
{"id": 19, "text": "Quero automatizar os agendamentos da minha clínica veterinária", "label": "hot"},
|
| 21 |
+
{"id": 20, "text": "Meu delivery precisa de cardápio digital, quero contratar", "label": "hot"},
|
| 22 |
+
{"id": 21, "text": "Quero começar esse mês, como faço pra assinar?", "label": "hot"},
|
| 23 |
+
{"id": 22, "text": "Quero contratar o serviço de marketing pra minha padaria", "label": "hot"},
|
| 24 |
+
{"id": 23, "text": "Tenho interesse em contratar, me manda os valores dos planos", "label": "hot"},
|
| 25 |
+
{"id": 24, "text": "Quero um e-commerce pra minha loja de calçados", "label": "hot"},
|
| 26 |
+
{"id": 25, "text": "Preciso de um app pra minha academia, quero fechar", "label": "hot"},
|
| 27 |
+
{"id": 26, "text": "Quero modernizar minha loja e fechar com vocês", "label": "hot"},
|
| 28 |
+
{"id": 27, "text": "Me manda o contrato que eu quero assinar essa semana", "label": "hot"},
|
| 29 |
+
{"id": 28, "text": "Quero colocar meu restaurante no aplicativo de entregas", "label": "hot"},
|
| 30 |
+
{"id": 29, "text": "Quero ter presença digital, quero contratar agora", "label": "hot"},
|
| 31 |
+
{"id": 30, "text": "Estou pesquisando sistemas pra minha oficina mecânica", "label": "warm"},
|
| 32 |
+
{"id": 31, "text": "Meu salão precisa de agenda, estou comparando opções", "label": "warm"},
|
| 33 |
+
{"id": 32, "text": "Pode me explicar melhor como funciona a automação?", "label": "warm"},
|
| 34 |
+
{"id": 33, "text": "Estou pensando em modernizar minha loja no próximo ano", "label": "warm"},
|
| 35 |
+
{"id": 34, "text": "Quanto tempo leva pra implementar no meu negócio?", "label": "warm"},
|
| 36 |
+
{"id": 35, "text": "Queria saber os valores antes de decidir", "label": "warm"},
|
| 37 |
+
{"id": 36, "text": "Meu restaurante tá crescendo e preciso me organizar melhor", "label": "warm"},
|
| 38 |
+
{"id": 37, "text": "Ouvi falar do serviço, quero conhecer melhor", "label": "warm"},
|
| 39 |
+
{"id": 38, "text": "Estou juntando propostas pra minha clínica", "label": "warm"},
|
| 40 |
+
{"id": 39, "text": "Ainda estou vendo se vale a pena investir", "label": "warm"},
|
| 41 |
+
{"id": 40, "text": "Meu sócio falou pra eu procurar vocês", "label": "warm"},
|
| 42 |
+
{"id": 41, "text": "Quero melhorar o atendimento do meu petshop", "label": "warm"},
|
| 43 |
+
{"id": 42, "text": "Como funciona pra quem tem duas lojas?", "label": "warm"},
|
| 44 |
+
{"id": 43, "text": "Tenho uma papelaria e queria saber mais", "label": "warm"},
|
| 45 |
+
{"id": 44, "text": "Dá pra integrar com o sistema que já uso?", "label": "warm"},
|
| 46 |
+
{"id": 45, "text": "Só estou olhando por curiosidade, sem compromisso", "label": "cold"},
|
| 47 |
+
{"id": 46, "text": "Achei interessante mas é só curiosidade", "label": "cold"},
|
| 48 |
+
{"id": 47, "text": "Talvez no futuro, obrigado", "label": "cold"},
|
| 49 |
+
{"id": 48, "text": "Quero só entender a plataforma por enquanto", "label": "cold"},
|
| 50 |
+
{"id": 49, "text": "Estou vendo preços no mercado, sem pressa", "label": "cold"},
|
| 51 |
+
{"id": 50, "text": "Recebi a mensagem no grupo, só vim ver", "label": "cold"},
|
| 52 |
+
{"id": 51, "text": "Não estou precisando no momento", "label": "cold"},
|
| 53 |
+
{"id": 52, "text": "Queria só saber o que o app faz", "label": "cold"},
|
| 54 |
+
{"id": 53, "text": "Meu negócio é pequeno demais pra isso agora", "label": "cold"},
|
| 55 |
+
{"id": 54, "text": "Só queria guardar o contato pra depois", "label": "cold"},
|
| 56 |
+
{"id": 55, "text": "Tô explorando, não sei nem se preciso", "label": "cold"},
|
| 57 |
+
{"id": 56, "text": "Curiosidade mesmo, pode me mandar o material", "label": "cold"},
|
| 58 |
+
{"id": 57, "text": "Vi o anúncio, mas não tenho interesse agora", "label": "cold"},
|
| 59 |
+
{"id": 58, "text": "Só queria saber o preço, mas não pretendo contratar", "label": "cold"},
|
| 60 |
+
{"id": 59, "text": "Vim só dar uma olhada", "label": "cold"},
|
| 61 |
+
{"id": 60, "text": "Sem interesse por enquanto, valeu", "label": "cold"}
|
| 62 |
+
]
|
experiment.py
ADDED
|
@@ -0,0 +1,340 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import csv
|
| 4 |
+
import json
|
| 5 |
+
import statistics
|
| 6 |
+
from dataclasses import dataclass, asdict
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
from typing import Iterable
|
| 9 |
+
|
| 10 |
+
import numpy as np
|
| 11 |
+
from sklearn.feature_extraction.text import TfidfVectorizer
|
| 12 |
+
from sklearn.linear_model import LogisticRegression
|
| 13 |
+
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
|
| 14 |
+
from sklearn.model_selection import StratifiedKFold, train_test_split
|
| 15 |
+
from sklearn.pipeline import FeatureUnion
|
| 16 |
+
|
| 17 |
+
DATA_PATH = Path(__file__).resolve().parents[2] / "data" / "sample_leads.json"
|
| 18 |
+
MODEL_DIR = Path(__file__).resolve().parents[2] / "model"
|
| 19 |
+
RUNS_DIR = Path(__file__).resolve().parents[2] / "runs"
|
| 20 |
+
|
| 21 |
+
CV_FOLDS = 5
|
| 22 |
+
RANDOM_STATE = 42
|
| 23 |
+
C_GRID = [0.5, 1.0, 2.0, 4.0]
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def _make_features() -> FeatureUnion:
|
| 27 |
+
"""Uniao de features: word n-grams (sintaxe) + char_wb n-grams (morfologia
|
| 28 |
+
do PT-BR — pega radical de 'contrat/automat/pesquis' sem depender de stemmer).
|
| 29 |
+
"""
|
| 30 |
+
return FeatureUnion(
|
| 31 |
+
[
|
| 32 |
+
("word", TfidfVectorizer(ngram_range=(1, 2), min_df=1, sublinear_tf=True)),
|
| 33 |
+
("char", TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4), min_df=1, sublinear_tf=True)),
|
| 34 |
+
]
|
| 35 |
+
)
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _make_clf(C: float = 1.0) -> LogisticRegression:
|
| 39 |
+
return LogisticRegression(max_iter=2000, class_weight="balanced", random_state=RANDOM_STATE, C=C)
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def _tune_c(texts: list[str], labels: list[str]) -> tuple[float, dict]:
|
| 43 |
+
"""Grid manual de C com CV interno — usado DENTRO do fold de treino
|
| 44 |
+
(nested CV): o tuning nunca ve o fold de teste.
|
| 45 |
+
"""
|
| 46 |
+
best_c, best_score = 1.0, -1.0
|
| 47 |
+
scores: dict[str, float] = {}
|
| 48 |
+
inner = StratifiedKFold(n_splits=3, shuffle=True, random_state=RANDOM_STATE)
|
| 49 |
+
for c in C_GRID:
|
| 50 |
+
fold_scores = []
|
| 51 |
+
for tr, va in inner.split(texts, labels):
|
| 52 |
+
vec = _make_features()
|
| 53 |
+
X_tr = vec.fit_transform([texts[i] for i in tr])
|
| 54 |
+
X_va = vec.transform([texts[i] for i in va])
|
| 55 |
+
clf = _make_clf(c)
|
| 56 |
+
clf.fit(X_tr, [labels[i] for i in tr])
|
| 57 |
+
_, _, f1, _ = precision_recall_fscore_support(
|
| 58 |
+
[labels[i] for i in va], clf.predict(X_va), average="macro", zero_division=0
|
| 59 |
+
)
|
| 60 |
+
fold_scores.append(float(f1))
|
| 61 |
+
mean = float(np.mean(fold_scores))
|
| 62 |
+
scores[str(c)] = round(mean, 4)
|
| 63 |
+
if mean > best_score:
|
| 64 |
+
best_c, best_score = c, mean
|
| 65 |
+
return best_c, scores
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
@dataclass
|
| 69 |
+
class Lead:
|
| 70 |
+
id: int
|
| 71 |
+
text: str
|
| 72 |
+
label: str
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
@dataclass
|
| 76 |
+
class RunRecord:
|
| 77 |
+
id: int
|
| 78 |
+
text: str
|
| 79 |
+
expected: str
|
| 80 |
+
predicted: str
|
| 81 |
+
correct: bool = False
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def _load_leads() -> list[Lead]:
|
| 85 |
+
with open(DATA_PATH, "r", encoding="utf-8") as f:
|
| 86 |
+
return [Lead(**item) for item in json.load(f)]
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def _preprocess(text: str) -> str:
|
| 90 |
+
return text.lower()
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def _make_pipeline():
|
| 94 |
+
return (
|
| 95 |
+
TfidfVectorizer(ngram_range=(1, 2), min_df=1),
|
| 96 |
+
LogisticRegression(max_iter=1000, class_weight="balanced", random_state=RANDOM_STATE),
|
| 97 |
+
)
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def train_model(leads: list[Lead]):
|
| 101 |
+
"""Treina o artefato final em TODOS os dados (uso em producao/demo).
|
| 102 |
+
|
| 103 |
+
O C vem de grid CV em dados completos; a avaliacao honesta NAO usa este
|
| 104 |
+
modelo — ver `cross_validate` (tuning aninhado dentro de cada fold).
|
| 105 |
+
"""
|
| 106 |
+
texts = [_preprocess(l.text) for l in leads]
|
| 107 |
+
labels = [l.label for l in leads]
|
| 108 |
+
|
| 109 |
+
best_c, _ = _tune_c(texts, labels)
|
| 110 |
+
vectorizer = _make_features()
|
| 111 |
+
X = vectorizer.fit_transform(texts)
|
| 112 |
+
clf = _make_clf(best_c)
|
| 113 |
+
clf.fit(X, labels)
|
| 114 |
+
clf.C_TUNED = best_c # type: ignore[attr-defined]
|
| 115 |
+
|
| 116 |
+
return vectorizer, clf
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def evaluate(vectorizer, clf, leads: list[Lead]) -> tuple[list[RunRecord], dict]:
|
| 120 |
+
"""Avaliacao in-sample (smoke test). NAO usar como metrica publicada:
|
| 121 |
+
aqui o modelo ve os mesmos exemplos no treino e na avaliacao.
|
| 122 |
+
"""
|
| 123 |
+
texts = [_preprocess(l.text) for l in leads]
|
| 124 |
+
X = vectorizer.transform(texts)
|
| 125 |
+
preds = clf.predict(X)
|
| 126 |
+
|
| 127 |
+
records: list[RunRecord] = []
|
| 128 |
+
for lead, pred in zip(leads, preds):
|
| 129 |
+
records.append(
|
| 130 |
+
RunRecord(id=lead.id, text=lead.text, expected=lead.label, predicted=str(pred), correct=str(pred) == lead.label)
|
| 131 |
+
)
|
| 132 |
+
|
| 133 |
+
acc = accuracy_score([r.expected for r in records], [r.predicted for r in records])
|
| 134 |
+
precision, recall, f1, _ = precision_recall_fscore_support(
|
| 135 |
+
[r.expected for r in records], [r.predicted for r in records], average="weighted", zero_division=0
|
| 136 |
+
)
|
| 137 |
+
|
| 138 |
+
summary = {
|
| 139 |
+
"accuracy": round(float(acc), 4),
|
| 140 |
+
"precision": round(float(precision), 4),
|
| 141 |
+
"recall": round(float(recall), 4),
|
| 142 |
+
"f1": round(float(f1), 4),
|
| 143 |
+
"samples": len(records),
|
| 144 |
+
"mode": "in-sample",
|
| 145 |
+
}
|
| 146 |
+
return records, summary
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def cross_validate(leads: list[Lead], k: int = CV_FOLDS) -> tuple[list[RunRecord], dict, list[dict], list[dict]]:
|
| 150 |
+
"""Avaliacao honesta: StratifiedKFold com vectorizer treinado SOMENTE
|
| 151 |
+
no fold de treino (sem vazamento). Retorna predicoes pooled out-of-fold,
|
| 152 |
+
resumo com media + desvio por fold, linhas por fold e relatorio por classe.
|
| 153 |
+
"""
|
| 154 |
+
texts = [_preprocess(l.text) for l in leads]
|
| 155 |
+
labels = [l.label for l in leads]
|
| 156 |
+
classes = sorted(set(labels))
|
| 157 |
+
|
| 158 |
+
skf = StratifiedKFold(n_splits=k, shuffle=True, random_state=RANDOM_STATE)
|
| 159 |
+
pooled: list[RunRecord] = []
|
| 160 |
+
fold_rows: list[dict] = []
|
| 161 |
+
fold_weighted: list[dict] = []
|
| 162 |
+
tuned_cs: list[float] = []
|
| 163 |
+
|
| 164 |
+
for fold, (train_idx, test_idx) in enumerate(skf.split(texts, labels), start=1):
|
| 165 |
+
train_texts = [texts[i] for i in train_idx]
|
| 166 |
+
train_labels = [labels[i] for i in train_idx]
|
| 167 |
+
|
| 168 |
+
# tuning aninhado: C escolhido só com o fold de treino
|
| 169 |
+
best_c, _ = _tune_c(train_texts, train_labels)
|
| 170 |
+
tuned_cs.append(best_c)
|
| 171 |
+
|
| 172 |
+
vectorizer = _make_features()
|
| 173 |
+
X_train = vectorizer.fit_transform(train_texts)
|
| 174 |
+
clf = _make_clf(best_c)
|
| 175 |
+
clf.fit(X_train, train_labels)
|
| 176 |
+
preds = clf.predict(vectorizer.transform([texts[i] for i in test_idx]))
|
| 177 |
+
|
| 178 |
+
for idx, pred in zip(test_idx, preds):
|
| 179 |
+
pooled.append(
|
| 180 |
+
RunRecord(id=leads[idx].id, text=leads[idx].text, expected=labels[idx], predicted=str(pred))
|
| 181 |
+
)
|
| 182 |
+
|
| 183 |
+
y_true = [labels[i] for i in test_idx]
|
| 184 |
+
precision, recall, f1, _ = precision_recall_fscore_support(
|
| 185 |
+
y_true, preds, average="weighted", zero_division=0
|
| 186 |
+
)
|
| 187 |
+
fold_weighted.append(
|
| 188 |
+
{
|
| 189 |
+
"accuracy": float(accuracy_score(y_true, preds)),
|
| 190 |
+
"precision": float(precision),
|
| 191 |
+
"recall": float(recall),
|
| 192 |
+
"f1": float(f1),
|
| 193 |
+
}
|
| 194 |
+
)
|
| 195 |
+
fold_rows.append(
|
| 196 |
+
{
|
| 197 |
+
"fold": fold,
|
| 198 |
+
"train": len(train_idx),
|
| 199 |
+
"test": len(test_idx),
|
| 200 |
+
"accuracy": round(fold_weighted[-1]["accuracy"], 4),
|
| 201 |
+
"precision": round(fold_weighted[-1]["precision"], 4),
|
| 202 |
+
"recall": round(fold_weighted[-1]["recall"], 4),
|
| 203 |
+
"f1": round(fold_weighted[-1]["f1"], 4),
|
| 204 |
+
}
|
| 205 |
+
)
|
| 206 |
+
|
| 207 |
+
y_true = [r.expected for r in pooled]
|
| 208 |
+
y_pred = [r.predicted for r in pooled]
|
| 209 |
+
|
| 210 |
+
pooled_precision, pooled_recall, pooled_f1, support = precision_recall_fscore_support(
|
| 211 |
+
y_true, y_pred, labels=classes, zero_division=0
|
| 212 |
+
)
|
| 213 |
+
macro_precision, macro_recall, macro_f1, _ = precision_recall_fscore_support(
|
| 214 |
+
y_true, y_pred, average="macro", zero_division=0
|
| 215 |
+
)
|
| 216 |
+
|
| 217 |
+
class_rows = [
|
| 218 |
+
{
|
| 219 |
+
"class": cls,
|
| 220 |
+
"precision": round(float(p), 4),
|
| 221 |
+
"recall": round(float(r), 4),
|
| 222 |
+
"f1": round(float(f), 4),
|
| 223 |
+
"support": int(s),
|
| 224 |
+
}
|
| 225 |
+
for cls, p, r, f, s in zip(classes, pooled_precision, pooled_recall, pooled_f1, support)
|
| 226 |
+
]
|
| 227 |
+
|
| 228 |
+
def _mean_std(key: str) -> tuple[float, float]:
|
| 229 |
+
values = [row[key] for row in fold_weighted]
|
| 230 |
+
return float(statistics.mean(values)), float(statistics.stdev(values) if len(values) > 1 else 0.0)
|
| 231 |
+
|
| 232 |
+
mean_acc, std_acc = _mean_std("accuracy")
|
| 233 |
+
mean_p, std_p = _mean_std("precision")
|
| 234 |
+
mean_r, std_r = _mean_std("recall")
|
| 235 |
+
mean_f1, std_f1 = _mean_std("f1")
|
| 236 |
+
|
| 237 |
+
summary = {
|
| 238 |
+
"mode": f"cv{k}",
|
| 239 |
+
"accuracy": round(mean_acc, 4),
|
| 240 |
+
"accuracy_std": round(std_acc, 4),
|
| 241 |
+
"precision": round(mean_p, 4),
|
| 242 |
+
"precision_std": round(std_p, 4),
|
| 243 |
+
"recall": round(mean_r, 4),
|
| 244 |
+
"recall_std": round(std_r, 4),
|
| 245 |
+
"f1": round(mean_f1, 4),
|
| 246 |
+
"f1_std": round(std_f1, 4),
|
| 247 |
+
"macro_f1": round(float(macro_f1), 4),
|
| 248 |
+
"folds": k,
|
| 249 |
+
"samples": len(pooled),
|
| 250 |
+
"c_per_fold": ";".join(str(c) for c in tuned_cs),
|
| 251 |
+
"_fold_rows": fold_rows,
|
| 252 |
+
"_class_rows": class_rows,
|
| 253 |
+
}
|
| 254 |
+
return pooled, summary, fold_rows, class_rows
|
| 255 |
+
|
| 256 |
+
|
| 257 |
+
def save_runs(records: list[RunRecord], summary: dict, runs_dir: str = "runs") -> tuple[Path, Path]:
|
| 258 |
+
runs = Path(runs_dir)
|
| 259 |
+
runs.mkdir(parents=True, exist_ok=True)
|
| 260 |
+
|
| 261 |
+
latest_path = runs / "latest.csv"
|
| 262 |
+
summary_path = runs / "summary.csv"
|
| 263 |
+
|
| 264 |
+
with open(latest_path, "w", newline="", encoding="utf-8") as f:
|
| 265 |
+
writer = csv.DictWriter(f, fieldnames=list(RunRecord.__dataclass_fields__.keys()))
|
| 266 |
+
writer.writeheader()
|
| 267 |
+
for r in records:
|
| 268 |
+
writer.writerow(asdict(r))
|
| 269 |
+
|
| 270 |
+
detail = {k: summary.pop(k) for k in list(summary) if k.startswith("_")}
|
| 271 |
+
with open(summary_path, "w", newline="", encoding="utf-8") as f:
|
| 272 |
+
writer = csv.DictWriter(f, fieldnames=list(summary.keys()))
|
| 273 |
+
writer.writeheader()
|
| 274 |
+
writer.writerow(summary)
|
| 275 |
+
|
| 276 |
+
for key, name in (("_fold_rows", "cv_folds.csv"), ("_class_rows", "class_report.csv")):
|
| 277 |
+
if key in detail:
|
| 278 |
+
with open(runs / name, "w", newline="", encoding="utf-8") as f:
|
| 279 |
+
writer = csv.DictWriter(f, fieldnames=list(detail[key][0].keys()))
|
| 280 |
+
writer.writeheader()
|
| 281 |
+
writer.writerows(detail[key])
|
| 282 |
+
|
| 283 |
+
return latest_path, summary_path
|
| 284 |
+
|
| 285 |
+
|
| 286 |
+
def predict_text(vectorizer, clf, text: str) -> str:
|
| 287 |
+
X = vectorizer.transform([_preprocess(text)])
|
| 288 |
+
return str(clf.predict(X)[0])
|
| 289 |
+
|
| 290 |
+
|
| 291 |
+
def _demo_predict(text: str) -> str:
|
| 292 |
+
t = text.lower()
|
| 293 |
+
if any(w in t for w in ["quero contratar", "quero automatizar", "preciso de um site", "quero um aplicativo", "quero contratar um personal"]):
|
| 294 |
+
return "hot"
|
| 295 |
+
if any(w in t for w in ["estou procurando", "quero melhorar", "precisa de um sistema", "precisa de ajuda", "começando"]):
|
| 296 |
+
return "warm"
|
| 297 |
+
return "cold"
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
def run_experiment(demo: bool = False, k: int = CV_FOLDS) -> tuple[list[RunRecord], dict]:
|
| 301 |
+
"""Fluxo principal.
|
| 302 |
+
|
| 303 |
+
demo=True: classificacao por regras (sem treino).
|
| 304 |
+
demo=False: treina o artefato final em todos os dados E publica as metricas
|
| 305 |
+
honestas de validacao cruzada estratificada (out-of-fold) em runs/.
|
| 306 |
+
"""
|
| 307 |
+
leads = _load_leads()
|
| 308 |
+
MODEL_DIR.mkdir(parents=True, exist_ok=True)
|
| 309 |
+
RUNS_DIR.mkdir(parents=True, exist_ok=True)
|
| 310 |
+
|
| 311 |
+
if demo:
|
| 312 |
+
records = []
|
| 313 |
+
for l in leads:
|
| 314 |
+
pred = _demo_predict(l.text)
|
| 315 |
+
records.append(
|
| 316 |
+
RunRecord(id=l.id, text=l.text, expected=l.label, predicted=pred, correct=pred == l.label)
|
| 317 |
+
)
|
| 318 |
+
acc = sum(1 for r in records if r.correct) / max(len(records), 1)
|
| 319 |
+
precision, recall, f1, _ = precision_recall_fscore_support(
|
| 320 |
+
[r.expected for r in records], [r.predicted for r in records], average="weighted", zero_division=0
|
| 321 |
+
)
|
| 322 |
+
summary = {
|
| 323 |
+
"accuracy": round(float(acc), 4),
|
| 324 |
+
"precision": round(float(precision), 4),
|
| 325 |
+
"recall": round(float(recall), 4),
|
| 326 |
+
"f1": round(float(f1), 4),
|
| 327 |
+
"samples": len(records),
|
| 328 |
+
"mode": "demo",
|
| 329 |
+
}
|
| 330 |
+
return records, summary
|
| 331 |
+
|
| 332 |
+
import joblib
|
| 333 |
+
|
| 334 |
+
vectorizer, clf = train_model(leads)
|
| 335 |
+
joblib.dump(vectorizer, MODEL_DIR / "vectorizer.pkl")
|
| 336 |
+
joblib.dump(clf, MODEL_DIR / "model.pkl")
|
| 337 |
+
|
| 338 |
+
records, summary, _, _ = cross_validate(leads, k=k)
|
| 339 |
+
summary["final_C"] = float(getattr(clf, "C_TUNED", clf.C))
|
| 340 |
+
return records, summary
|
model.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1b59eb40aaab924e5fa4d37cc6ccd31b457b6f9ec976193ab48974dcaf0bc319
|
| 3 |
+
size 51586
|
runs/class_report.csv
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
class,precision,recall,f1,support
|
| 2 |
+
cold,0.7,0.7,0.7,20
|
| 3 |
+
hot,0.75,0.75,0.75,20
|
| 4 |
+
warm,0.45,0.45,0.45,20
|
runs/cv_folds.csv
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
fold,train,test,accuracy,precision,recall,f1
|
| 2 |
+
1,48,12,0.5,0.5,0.5,0.5
|
| 3 |
+
2,48,12,0.5,0.5,0.5,0.5
|
| 4 |
+
3,48,12,0.75,0.7389,0.75,0.7368
|
| 5 |
+
4,48,12,0.9167,0.9333,0.9167,0.9153
|
| 6 |
+
5,48,12,0.5,0.5,0.5,0.5
|
runs/summary.csv
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
mode,accuracy,accuracy_std,precision,precision_std,recall,recall_std,f1,f1_std,macro_f1,folds,samples,c_per_fold,final_C
|
| 2 |
+
cv5,0.6333,0.1918,0.6344,0.1965,0.6333,0.1918,0.6304,0.1894,0.6333,5,60,2.0;0.5;0.5;2.0;0.5,4.0
|
vectorizer.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b089dbf24a56312e477d0f44ce3e0cf579995b3f2cb308e243329b75429c6c84
|
| 3 |
+
size 40357
|