ranzouu commited on
Commit
8772612
·
verified ·
1 Parent(s): 195e446

Upload folder using huggingface_hub

Browse files
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Rayson Severiano
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: sklearn
3
+ language: pt
4
+ license: mit
5
+ pipeline_tag: text-classification
6
+ tags:
7
+ - text-classification
8
+ - intent-classification
9
+ - lead-scoring
10
+ - tf-idf
11
+ - logistic-regression
12
+ - portuguese
13
+ ---
14
+
15
+ # LeadForge Intent Classifier
16
+
17
+ Classificador de **intenção de prospecção** (`hot` / `warm` / `cold`) a partir do texto do lead, em português. Pipeline TF-IDF (word n-grams + char_wb n-grams) + Logistic Regression balanceada — sem API paga, sem GPU, determinístico (`random_state=42`).
18
+
19
+ Faz parte do pipeline do [LeadForge](https://github.com/ranzouu/leadforge). Código completo, CLI e testes: [github.com/ranzouu/leadforge-intent-classifier](https://github.com/ranzouu/leadforge-intent-classifier).
20
+
21
+ ## Métricas — avaliação honesta (out-of-fold)
22
+
23
+ **StratifiedKFold (5 folds) com tuning de `C` aninhado dentro do fold de treino.** Os números são pooled out-of-fold: toda amostra foi classificada por um modelo que nunca a viu no treino.
24
+
25
+ | Métrica (média ± desvio entre folds) | Valor |
26
+ |---|---|
27
+ | Accuracy | 0.633 ± 0.192 |
28
+ | Precision (weighted) | 0.634 ± 0.197 |
29
+ | Recall (weighted) | 0.633 ± 0.192 |
30
+ | F1 (weighted) | 0.630 ± 0.189 |
31
+ | F1 (macro) | 0.633 |
32
+
33
+ | Classe | Precision | Recall | F1 | Suporte |
34
+ |---|---|---|---|---|
35
+ | cold | 0.700 | 0.700 | 0.700 | 20 |
36
+ | hot | 0.750 | 0.750 | 0.750 | 20 |
37
+ | warm | 0.450 | 0.450 | 0.450 | 20 |
38
+
39
+ > Nota de integridade: uma versão anterior deste projeto publicava "accuracy 1.0" medida no mesmo conjunto usado no treino (avaliação in-sample — inflação). Esse número foi removido e substituído pela validação cruzada acima. Detalhes por fold em `runs/cv_folds.csv` e por classe em `runs/class_report.csv`.
40
+
41
+ ## Uso
42
+
43
+ ```python
44
+ import joblib
45
+
46
+ vectorizer = joblib.load("vectorizer.pkl")
47
+ clf = joblib.load("model.pkl")
48
+
49
+ frases = [
50
+ "Quero fechar um plano hoje para minha barbearia", # hot
51
+ "Pode me explicar melhor como funciona?", # warm
52
+ "Só estou olhando por curiosidade", # cold
53
+ ]
54
+ print(clf.predict(vectorizer.transform([f.lower() for f in frases])))
55
+ ```
56
+
57
+ ## Limitações (lé com atenção)
58
+
59
+ - **Dataset de demonstração**: 60 amostras (20 por classe) escritas para exercitar o pipeline — não são leads reais de produção. Em produção, treinar com dados reais e revalidar antes de confiar.
60
+ - **`warm` é a fronteira difícil** (F1 0.45): separar "interesse" de "só curiosidade" é tênue até em leitura humana.
61
+ - Modelo linha de base (TF-IDF + LR); com dados reais e volume maior, comparar com alternativas (LinearSVC, gradient boosting, LLM zero-shot) antes de escolher.
62
+
63
+ ## Reproduzir
64
+
65
+ ```bash
66
+ pip install -e . # do repositório
67
+ python -m leadforge_intent_classifier
68
+ # runs/summary.csv, runs/cv_folds.csv, runs/class_report.csv, runs/latest.csv
69
+ ```
70
+
71
+ ## Arquivos
72
+
73
+ | Arquivo | Conteúdo |
74
+ |---|---|
75
+ | `model.pkl` | Logistic Regression final (treinada nos 60 samples, `C=4.0`) |
76
+ | `vectorizer.pkl` | FeatureUnion word(1,2) + char_wb(2,4), `sublinear_tf` |
77
+ | `experiment.py` | código-fonte do pipeline (treino, nested CV, exportação) |
78
+ | `data/sample_leads.json` | dataset de demo (60 amostras) |
79
+ | `runs/*.csv` | métricas por fold, por classe e predições out-of-fold |
80
+ | `LICENSE` | MIT |
data/sample_leads.json ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {"id": 1, "text": "Quero contratar um plano de musculação para minha academia", "label": "hot"},
3
+ {"id": 2, "text": "Preciso de um site para meu petshop", "label": "hot"},
4
+ {"id": 3, "text": "Quero automatizar o cardápio do meu restaurante", "label": "hot"},
5
+ {"id": 4, "text": "Estou procurando um sistema para minha loja", "label": "warm"},
6
+ {"id": 5, "text": "Quero melhorar a organização da minha academia", "label": "warm"},
7
+ {"id": 6, "text": "Meu petshop precisa de um sistema de agenda", "label": "warm"},
8
+ {"id": 7, "text": "Quero entender mais sobre automação", "label": "cold"},
9
+ {"id": 8, "text": "Estou apenas pesquisando opções", "label": "cold"},
10
+ {"id": 9, "text": "Quero um aplicativo para meu negócio", "label": "hot"},
11
+ {"id": 10, "text": "Preciso de ajuda com marketing digital", "label": "warm"},
12
+ {"id": 11, "text": "Só queria ver como funciona", "label": "cold"},
13
+ {"id": 12, "text": "Quero contratar um personal para minha academia", "label": "hot"},
14
+ {"id": 13, "text": "Estou começando um petshop e preciso de ajuda", "label": "warm"},
15
+ {"id": 14, "text": "Quero criar um site institucional", "label": "hot"},
16
+ {"id": 15, "text": "Ainda não sei se preciso de um sistema", "label": "cold"},
17
+ {"id": 16, "text": "Quero fechar um plano hoje para minha barbearia", "label": "hot"},
18
+ {"id": 17, "text": "Preciso contratar um sistema pro meu salão essa semana", "label": "hot"},
19
+ {"id": 18, "text": "Quero um site pra minha loja de roupas, pode mandar proposta", "label": "hot"},
20
+ {"id": 19, "text": "Quero automatizar os agendamentos da minha clínica veterinária", "label": "hot"},
21
+ {"id": 20, "text": "Meu delivery precisa de cardápio digital, quero contratar", "label": "hot"},
22
+ {"id": 21, "text": "Quero começar esse mês, como faço pra assinar?", "label": "hot"},
23
+ {"id": 22, "text": "Quero contratar o serviço de marketing pra minha padaria", "label": "hot"},
24
+ {"id": 23, "text": "Tenho interesse em contratar, me manda os valores dos planos", "label": "hot"},
25
+ {"id": 24, "text": "Quero um e-commerce pra minha loja de calçados", "label": "hot"},
26
+ {"id": 25, "text": "Preciso de um app pra minha academia, quero fechar", "label": "hot"},
27
+ {"id": 26, "text": "Quero modernizar minha loja e fechar com vocês", "label": "hot"},
28
+ {"id": 27, "text": "Me manda o contrato que eu quero assinar essa semana", "label": "hot"},
29
+ {"id": 28, "text": "Quero colocar meu restaurante no aplicativo de entregas", "label": "hot"},
30
+ {"id": 29, "text": "Quero ter presença digital, quero contratar agora", "label": "hot"},
31
+ {"id": 30, "text": "Estou pesquisando sistemas pra minha oficina mecânica", "label": "warm"},
32
+ {"id": 31, "text": "Meu salão precisa de agenda, estou comparando opções", "label": "warm"},
33
+ {"id": 32, "text": "Pode me explicar melhor como funciona a automação?", "label": "warm"},
34
+ {"id": 33, "text": "Estou pensando em modernizar minha loja no próximo ano", "label": "warm"},
35
+ {"id": 34, "text": "Quanto tempo leva pra implementar no meu negócio?", "label": "warm"},
36
+ {"id": 35, "text": "Queria saber os valores antes de decidir", "label": "warm"},
37
+ {"id": 36, "text": "Meu restaurante tá crescendo e preciso me organizar melhor", "label": "warm"},
38
+ {"id": 37, "text": "Ouvi falar do serviço, quero conhecer melhor", "label": "warm"},
39
+ {"id": 38, "text": "Estou juntando propostas pra minha clínica", "label": "warm"},
40
+ {"id": 39, "text": "Ainda estou vendo se vale a pena investir", "label": "warm"},
41
+ {"id": 40, "text": "Meu sócio falou pra eu procurar vocês", "label": "warm"},
42
+ {"id": 41, "text": "Quero melhorar o atendimento do meu petshop", "label": "warm"},
43
+ {"id": 42, "text": "Como funciona pra quem tem duas lojas?", "label": "warm"},
44
+ {"id": 43, "text": "Tenho uma papelaria e queria saber mais", "label": "warm"},
45
+ {"id": 44, "text": "Dá pra integrar com o sistema que já uso?", "label": "warm"},
46
+ {"id": 45, "text": "Só estou olhando por curiosidade, sem compromisso", "label": "cold"},
47
+ {"id": 46, "text": "Achei interessante mas é só curiosidade", "label": "cold"},
48
+ {"id": 47, "text": "Talvez no futuro, obrigado", "label": "cold"},
49
+ {"id": 48, "text": "Quero só entender a plataforma por enquanto", "label": "cold"},
50
+ {"id": 49, "text": "Estou vendo preços no mercado, sem pressa", "label": "cold"},
51
+ {"id": 50, "text": "Recebi a mensagem no grupo, só vim ver", "label": "cold"},
52
+ {"id": 51, "text": "Não estou precisando no momento", "label": "cold"},
53
+ {"id": 52, "text": "Queria só saber o que o app faz", "label": "cold"},
54
+ {"id": 53, "text": "Meu negócio é pequeno demais pra isso agora", "label": "cold"},
55
+ {"id": 54, "text": "Só queria guardar o contato pra depois", "label": "cold"},
56
+ {"id": 55, "text": "Tô explorando, não sei nem se preciso", "label": "cold"},
57
+ {"id": 56, "text": "Curiosidade mesmo, pode me mandar o material", "label": "cold"},
58
+ {"id": 57, "text": "Vi o anúncio, mas não tenho interesse agora", "label": "cold"},
59
+ {"id": 58, "text": "Só queria saber o preço, mas não pretendo contratar", "label": "cold"},
60
+ {"id": 59, "text": "Vim só dar uma olhada", "label": "cold"},
61
+ {"id": 60, "text": "Sem interesse por enquanto, valeu", "label": "cold"}
62
+ ]
experiment.py ADDED
@@ -0,0 +1,340 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import csv
4
+ import json
5
+ import statistics
6
+ from dataclasses import dataclass, asdict
7
+ from pathlib import Path
8
+ from typing import Iterable
9
+
10
+ import numpy as np
11
+ from sklearn.feature_extraction.text import TfidfVectorizer
12
+ from sklearn.linear_model import LogisticRegression
13
+ from sklearn.metrics import accuracy_score, precision_recall_fscore_support
14
+ from sklearn.model_selection import StratifiedKFold, train_test_split
15
+ from sklearn.pipeline import FeatureUnion
16
+
17
+ DATA_PATH = Path(__file__).resolve().parents[2] / "data" / "sample_leads.json"
18
+ MODEL_DIR = Path(__file__).resolve().parents[2] / "model"
19
+ RUNS_DIR = Path(__file__).resolve().parents[2] / "runs"
20
+
21
+ CV_FOLDS = 5
22
+ RANDOM_STATE = 42
23
+ C_GRID = [0.5, 1.0, 2.0, 4.0]
24
+
25
+
26
+ def _make_features() -> FeatureUnion:
27
+ """Uniao de features: word n-grams (sintaxe) + char_wb n-grams (morfologia
28
+ do PT-BR — pega radical de 'contrat/automat/pesquis' sem depender de stemmer).
29
+ """
30
+ return FeatureUnion(
31
+ [
32
+ ("word", TfidfVectorizer(ngram_range=(1, 2), min_df=1, sublinear_tf=True)),
33
+ ("char", TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4), min_df=1, sublinear_tf=True)),
34
+ ]
35
+ )
36
+
37
+
38
+ def _make_clf(C: float = 1.0) -> LogisticRegression:
39
+ return LogisticRegression(max_iter=2000, class_weight="balanced", random_state=RANDOM_STATE, C=C)
40
+
41
+
42
+ def _tune_c(texts: list[str], labels: list[str]) -> tuple[float, dict]:
43
+ """Grid manual de C com CV interno — usado DENTRO do fold de treino
44
+ (nested CV): o tuning nunca ve o fold de teste.
45
+ """
46
+ best_c, best_score = 1.0, -1.0
47
+ scores: dict[str, float] = {}
48
+ inner = StratifiedKFold(n_splits=3, shuffle=True, random_state=RANDOM_STATE)
49
+ for c in C_GRID:
50
+ fold_scores = []
51
+ for tr, va in inner.split(texts, labels):
52
+ vec = _make_features()
53
+ X_tr = vec.fit_transform([texts[i] for i in tr])
54
+ X_va = vec.transform([texts[i] for i in va])
55
+ clf = _make_clf(c)
56
+ clf.fit(X_tr, [labels[i] for i in tr])
57
+ _, _, f1, _ = precision_recall_fscore_support(
58
+ [labels[i] for i in va], clf.predict(X_va), average="macro", zero_division=0
59
+ )
60
+ fold_scores.append(float(f1))
61
+ mean = float(np.mean(fold_scores))
62
+ scores[str(c)] = round(mean, 4)
63
+ if mean > best_score:
64
+ best_c, best_score = c, mean
65
+ return best_c, scores
66
+
67
+
68
+ @dataclass
69
+ class Lead:
70
+ id: int
71
+ text: str
72
+ label: str
73
+
74
+
75
+ @dataclass
76
+ class RunRecord:
77
+ id: int
78
+ text: str
79
+ expected: str
80
+ predicted: str
81
+ correct: bool = False
82
+
83
+
84
+ def _load_leads() -> list[Lead]:
85
+ with open(DATA_PATH, "r", encoding="utf-8") as f:
86
+ return [Lead(**item) for item in json.load(f)]
87
+
88
+
89
+ def _preprocess(text: str) -> str:
90
+ return text.lower()
91
+
92
+
93
+ def _make_pipeline():
94
+ return (
95
+ TfidfVectorizer(ngram_range=(1, 2), min_df=1),
96
+ LogisticRegression(max_iter=1000, class_weight="balanced", random_state=RANDOM_STATE),
97
+ )
98
+
99
+
100
+ def train_model(leads: list[Lead]):
101
+ """Treina o artefato final em TODOS os dados (uso em producao/demo).
102
+
103
+ O C vem de grid CV em dados completos; a avaliacao honesta NAO usa este
104
+ modelo — ver `cross_validate` (tuning aninhado dentro de cada fold).
105
+ """
106
+ texts = [_preprocess(l.text) for l in leads]
107
+ labels = [l.label for l in leads]
108
+
109
+ best_c, _ = _tune_c(texts, labels)
110
+ vectorizer = _make_features()
111
+ X = vectorizer.fit_transform(texts)
112
+ clf = _make_clf(best_c)
113
+ clf.fit(X, labels)
114
+ clf.C_TUNED = best_c # type: ignore[attr-defined]
115
+
116
+ return vectorizer, clf
117
+
118
+
119
+ def evaluate(vectorizer, clf, leads: list[Lead]) -> tuple[list[RunRecord], dict]:
120
+ """Avaliacao in-sample (smoke test). NAO usar como metrica publicada:
121
+ aqui o modelo ve os mesmos exemplos no treino e na avaliacao.
122
+ """
123
+ texts = [_preprocess(l.text) for l in leads]
124
+ X = vectorizer.transform(texts)
125
+ preds = clf.predict(X)
126
+
127
+ records: list[RunRecord] = []
128
+ for lead, pred in zip(leads, preds):
129
+ records.append(
130
+ RunRecord(id=lead.id, text=lead.text, expected=lead.label, predicted=str(pred), correct=str(pred) == lead.label)
131
+ )
132
+
133
+ acc = accuracy_score([r.expected for r in records], [r.predicted for r in records])
134
+ precision, recall, f1, _ = precision_recall_fscore_support(
135
+ [r.expected for r in records], [r.predicted for r in records], average="weighted", zero_division=0
136
+ )
137
+
138
+ summary = {
139
+ "accuracy": round(float(acc), 4),
140
+ "precision": round(float(precision), 4),
141
+ "recall": round(float(recall), 4),
142
+ "f1": round(float(f1), 4),
143
+ "samples": len(records),
144
+ "mode": "in-sample",
145
+ }
146
+ return records, summary
147
+
148
+
149
+ def cross_validate(leads: list[Lead], k: int = CV_FOLDS) -> tuple[list[RunRecord], dict, list[dict], list[dict]]:
150
+ """Avaliacao honesta: StratifiedKFold com vectorizer treinado SOMENTE
151
+ no fold de treino (sem vazamento). Retorna predicoes pooled out-of-fold,
152
+ resumo com media + desvio por fold, linhas por fold e relatorio por classe.
153
+ """
154
+ texts = [_preprocess(l.text) for l in leads]
155
+ labels = [l.label for l in leads]
156
+ classes = sorted(set(labels))
157
+
158
+ skf = StratifiedKFold(n_splits=k, shuffle=True, random_state=RANDOM_STATE)
159
+ pooled: list[RunRecord] = []
160
+ fold_rows: list[dict] = []
161
+ fold_weighted: list[dict] = []
162
+ tuned_cs: list[float] = []
163
+
164
+ for fold, (train_idx, test_idx) in enumerate(skf.split(texts, labels), start=1):
165
+ train_texts = [texts[i] for i in train_idx]
166
+ train_labels = [labels[i] for i in train_idx]
167
+
168
+ # tuning aninhado: C escolhido só com o fold de treino
169
+ best_c, _ = _tune_c(train_texts, train_labels)
170
+ tuned_cs.append(best_c)
171
+
172
+ vectorizer = _make_features()
173
+ X_train = vectorizer.fit_transform(train_texts)
174
+ clf = _make_clf(best_c)
175
+ clf.fit(X_train, train_labels)
176
+ preds = clf.predict(vectorizer.transform([texts[i] for i in test_idx]))
177
+
178
+ for idx, pred in zip(test_idx, preds):
179
+ pooled.append(
180
+ RunRecord(id=leads[idx].id, text=leads[idx].text, expected=labels[idx], predicted=str(pred))
181
+ )
182
+
183
+ y_true = [labels[i] for i in test_idx]
184
+ precision, recall, f1, _ = precision_recall_fscore_support(
185
+ y_true, preds, average="weighted", zero_division=0
186
+ )
187
+ fold_weighted.append(
188
+ {
189
+ "accuracy": float(accuracy_score(y_true, preds)),
190
+ "precision": float(precision),
191
+ "recall": float(recall),
192
+ "f1": float(f1),
193
+ }
194
+ )
195
+ fold_rows.append(
196
+ {
197
+ "fold": fold,
198
+ "train": len(train_idx),
199
+ "test": len(test_idx),
200
+ "accuracy": round(fold_weighted[-1]["accuracy"], 4),
201
+ "precision": round(fold_weighted[-1]["precision"], 4),
202
+ "recall": round(fold_weighted[-1]["recall"], 4),
203
+ "f1": round(fold_weighted[-1]["f1"], 4),
204
+ }
205
+ )
206
+
207
+ y_true = [r.expected for r in pooled]
208
+ y_pred = [r.predicted for r in pooled]
209
+
210
+ pooled_precision, pooled_recall, pooled_f1, support = precision_recall_fscore_support(
211
+ y_true, y_pred, labels=classes, zero_division=0
212
+ )
213
+ macro_precision, macro_recall, macro_f1, _ = precision_recall_fscore_support(
214
+ y_true, y_pred, average="macro", zero_division=0
215
+ )
216
+
217
+ class_rows = [
218
+ {
219
+ "class": cls,
220
+ "precision": round(float(p), 4),
221
+ "recall": round(float(r), 4),
222
+ "f1": round(float(f), 4),
223
+ "support": int(s),
224
+ }
225
+ for cls, p, r, f, s in zip(classes, pooled_precision, pooled_recall, pooled_f1, support)
226
+ ]
227
+
228
+ def _mean_std(key: str) -> tuple[float, float]:
229
+ values = [row[key] for row in fold_weighted]
230
+ return float(statistics.mean(values)), float(statistics.stdev(values) if len(values) > 1 else 0.0)
231
+
232
+ mean_acc, std_acc = _mean_std("accuracy")
233
+ mean_p, std_p = _mean_std("precision")
234
+ mean_r, std_r = _mean_std("recall")
235
+ mean_f1, std_f1 = _mean_std("f1")
236
+
237
+ summary = {
238
+ "mode": f"cv{k}",
239
+ "accuracy": round(mean_acc, 4),
240
+ "accuracy_std": round(std_acc, 4),
241
+ "precision": round(mean_p, 4),
242
+ "precision_std": round(std_p, 4),
243
+ "recall": round(mean_r, 4),
244
+ "recall_std": round(std_r, 4),
245
+ "f1": round(mean_f1, 4),
246
+ "f1_std": round(std_f1, 4),
247
+ "macro_f1": round(float(macro_f1), 4),
248
+ "folds": k,
249
+ "samples": len(pooled),
250
+ "c_per_fold": ";".join(str(c) for c in tuned_cs),
251
+ "_fold_rows": fold_rows,
252
+ "_class_rows": class_rows,
253
+ }
254
+ return pooled, summary, fold_rows, class_rows
255
+
256
+
257
+ def save_runs(records: list[RunRecord], summary: dict, runs_dir: str = "runs") -> tuple[Path, Path]:
258
+ runs = Path(runs_dir)
259
+ runs.mkdir(parents=True, exist_ok=True)
260
+
261
+ latest_path = runs / "latest.csv"
262
+ summary_path = runs / "summary.csv"
263
+
264
+ with open(latest_path, "w", newline="", encoding="utf-8") as f:
265
+ writer = csv.DictWriter(f, fieldnames=list(RunRecord.__dataclass_fields__.keys()))
266
+ writer.writeheader()
267
+ for r in records:
268
+ writer.writerow(asdict(r))
269
+
270
+ detail = {k: summary.pop(k) for k in list(summary) if k.startswith("_")}
271
+ with open(summary_path, "w", newline="", encoding="utf-8") as f:
272
+ writer = csv.DictWriter(f, fieldnames=list(summary.keys()))
273
+ writer.writeheader()
274
+ writer.writerow(summary)
275
+
276
+ for key, name in (("_fold_rows", "cv_folds.csv"), ("_class_rows", "class_report.csv")):
277
+ if key in detail:
278
+ with open(runs / name, "w", newline="", encoding="utf-8") as f:
279
+ writer = csv.DictWriter(f, fieldnames=list(detail[key][0].keys()))
280
+ writer.writeheader()
281
+ writer.writerows(detail[key])
282
+
283
+ return latest_path, summary_path
284
+
285
+
286
+ def predict_text(vectorizer, clf, text: str) -> str:
287
+ X = vectorizer.transform([_preprocess(text)])
288
+ return str(clf.predict(X)[0])
289
+
290
+
291
+ def _demo_predict(text: str) -> str:
292
+ t = text.lower()
293
+ if any(w in t for w in ["quero contratar", "quero automatizar", "preciso de um site", "quero um aplicativo", "quero contratar um personal"]):
294
+ return "hot"
295
+ if any(w in t for w in ["estou procurando", "quero melhorar", "precisa de um sistema", "precisa de ajuda", "começando"]):
296
+ return "warm"
297
+ return "cold"
298
+
299
+
300
+ def run_experiment(demo: bool = False, k: int = CV_FOLDS) -> tuple[list[RunRecord], dict]:
301
+ """Fluxo principal.
302
+
303
+ demo=True: classificacao por regras (sem treino).
304
+ demo=False: treina o artefato final em todos os dados E publica as metricas
305
+ honestas de validacao cruzada estratificada (out-of-fold) em runs/.
306
+ """
307
+ leads = _load_leads()
308
+ MODEL_DIR.mkdir(parents=True, exist_ok=True)
309
+ RUNS_DIR.mkdir(parents=True, exist_ok=True)
310
+
311
+ if demo:
312
+ records = []
313
+ for l in leads:
314
+ pred = _demo_predict(l.text)
315
+ records.append(
316
+ RunRecord(id=l.id, text=l.text, expected=l.label, predicted=pred, correct=pred == l.label)
317
+ )
318
+ acc = sum(1 for r in records if r.correct) / max(len(records), 1)
319
+ precision, recall, f1, _ = precision_recall_fscore_support(
320
+ [r.expected for r in records], [r.predicted for r in records], average="weighted", zero_division=0
321
+ )
322
+ summary = {
323
+ "accuracy": round(float(acc), 4),
324
+ "precision": round(float(precision), 4),
325
+ "recall": round(float(recall), 4),
326
+ "f1": round(float(f1), 4),
327
+ "samples": len(records),
328
+ "mode": "demo",
329
+ }
330
+ return records, summary
331
+
332
+ import joblib
333
+
334
+ vectorizer, clf = train_model(leads)
335
+ joblib.dump(vectorizer, MODEL_DIR / "vectorizer.pkl")
336
+ joblib.dump(clf, MODEL_DIR / "model.pkl")
337
+
338
+ records, summary, _, _ = cross_validate(leads, k=k)
339
+ summary["final_C"] = float(getattr(clf, "C_TUNED", clf.C))
340
+ return records, summary
model.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1b59eb40aaab924e5fa4d37cc6ccd31b457b6f9ec976193ab48974dcaf0bc319
3
+ size 51586
runs/class_report.csv ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ class,precision,recall,f1,support
2
+ cold,0.7,0.7,0.7,20
3
+ hot,0.75,0.75,0.75,20
4
+ warm,0.45,0.45,0.45,20
runs/cv_folds.csv ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ fold,train,test,accuracy,precision,recall,f1
2
+ 1,48,12,0.5,0.5,0.5,0.5
3
+ 2,48,12,0.5,0.5,0.5,0.5
4
+ 3,48,12,0.75,0.7389,0.75,0.7368
5
+ 4,48,12,0.9167,0.9333,0.9167,0.9153
6
+ 5,48,12,0.5,0.5,0.5,0.5
runs/summary.csv ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ mode,accuracy,accuracy_std,precision,precision_std,recall,recall_std,f1,f1_std,macro_f1,folds,samples,c_per_fold,final_C
2
+ cv5,0.6333,0.1918,0.6344,0.1965,0.6333,0.1918,0.6304,0.1894,0.6333,5,60,2.0;0.5;0.5;2.0;0.5,4.0
vectorizer.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b089dbf24a56312e477d0f44ce3e0cf579995b3f2cb308e243329b75429c6c84
3
+ size 40357