File size: 2,619 Bytes
b99688d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
language: [pt, en]
license: mit
pipeline_tag: text-classification
library_name: sentence-transformers
base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
datasets: [fernandofa42/architecture-decisions-pt-en]
tags: [software-architecture, adr, postmortem, documentation, portuguese, bilingual, logistic-regression]
metrics: [accuracy, f1]
---
# Architecture Doc Type Classifier (PT/EN)
Tells you whether an architecture document reads like an **ADR, design doc, guide, playbook, postmortem or teardown**, in Portuguese or English. It is a 6-class logistic regression on top of `paraphrase-multilingual-MiniLM-L12-v2`, with 2,310 weights, in plain JSON.
Use it to triage a documentation folder, not to decide anything alone. [Try it in the Space](https://huggingface.co/spaces/fernandofa42/architecture-search).
## How well it works
Measured with 5-fold cross-validation, always keeping the pt and en versions of a document in the same fold.
| Metric | Value |
|---|---|
| Accuracy | **0.648** (always guessing the largest class: 0.284) |
| Macro F1 | **0.643** |
| Class | F1 | Samples |
|---|---|---|
| postmortem | 0.92 | 40 |
| guide | 0.71 | 16 |
| teardown | 0.60 | 20 |
| playbook | 0.59 | 26 |
| adr | 0.55 | 50 |
| design-doc | 0.47 | 24 |
Postmortems are easy because they have a very particular structure. ADRs and design docs get confused with each other, which matches what happens in real life: many ADRs carry a design inside. Full numbers are in `metrics.json`.
## Limits
- **Small training set:** 176 samples from 88 documents.
- **AI-generated training data:** real documents from your company will look different, so expect lower numbers. I have not measured that.
- **Not robust to out-of-scope text:** it always answers one of the six classes, even for a recipe. Treat low confidence as "none of these".
- The label names themselves (`ADR`, `postmortem`...) are masked before training and inference, so the model learns from the content and not from the document announcing its type.
## Usage
```python
from huggingface_hub import hf_hub_download
import importlib.util
path = hf_hub_download("fernandofa42/architecture-doc-type-classifier", "inference.py")
spec = importlib.util.spec_from_file_location("inference", path)
inference = importlib.util.module_from_spec(spec)
spec.loader.exec_module(inference)
clf = inference.load()
print(clf.predict("Tivemos 40 minutos de indisponibilidade no PIX. Linha do tempo, causa raiz e ações corretivas..."))
```
Built by [Fernando Azevedo](https://fernando.moretes.com), Specialist IT Architect focused on AI and AWS.
|