| --- |
| language: [pt, en] |
| license: mit |
| pipeline_tag: text-classification |
| library_name: sentence-transformers |
| base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 |
| datasets: [fernandofa42/architecture-decisions-pt-en] |
| tags: [software-architecture, adr, postmortem, documentation, portuguese, bilingual, logistic-regression] |
| metrics: [accuracy, f1] |
| --- |
| |
| # Architecture Doc Type Classifier (PT/EN) |
|
|
| Tells you whether an architecture document reads like an **ADR, design doc, guide, playbook, postmortem or teardown**, in Portuguese or English. It is a 6-class logistic regression on top of `paraphrase-multilingual-MiniLM-L12-v2`, with 2,310 weights, in plain JSON. |
|
|
| Use it to triage a documentation folder, not to decide anything alone. [Try it in the Space](https://huggingface.co/spaces/fernandofa42/architecture-search). |
|
|
| ## How well it works |
|
|
| Measured with 5-fold cross-validation, always keeping the pt and en versions of a document in the same fold. |
|
|
| | Metric | Value | |
| |---|---| |
| | Accuracy | **0.648** (always guessing the largest class: 0.284) | |
| | Macro F1 | **0.643** | |
|
|
| | Class | F1 | Samples | |
| |---|---|---| |
| | postmortem | 0.92 | 40 | |
| | guide | 0.71 | 16 | |
| | teardown | 0.60 | 20 | |
| | playbook | 0.59 | 26 | |
| | adr | 0.55 | 50 | |
| | design-doc | 0.47 | 24 | |
|
|
| Postmortems are easy because they have a very particular structure. ADRs and design docs get confused with each other, which matches what happens in real life: many ADRs carry a design inside. Full numbers are in `metrics.json`. |
|
|
| ## Limits |
|
|
| - **Small training set:** 176 samples from 88 documents. |
| - **AI-generated training data:** real documents from your company will look different, so expect lower numbers. I have not measured that. |
| - **Not robust to out-of-scope text:** it always answers one of the six classes, even for a recipe. Treat low confidence as "none of these". |
| - The label names themselves (`ADR`, `postmortem`...) are masked before training and inference, so the model learns from the content and not from the document announcing its type. |
|
|
| ## Usage |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| import importlib.util |
| |
| path = hf_hub_download("fernandofa42/architecture-doc-type-classifier", "inference.py") |
| spec = importlib.util.spec_from_file_location("inference", path) |
| inference = importlib.util.module_from_spec(spec) |
| spec.loader.exec_module(inference) |
| |
| clf = inference.load() |
| print(clf.predict("Tivemos 40 minutos de indisponibilidade no PIX. Linha do tempo, causa raiz e ações corretivas...")) |
| ``` |
|
|
| Built by [Fernando Azevedo](https://fernando.moretes.com), Specialist IT Architect focused on AI and AWS. |
|
|
