propertypilot/property-pilot-triage

PropertyPilot Triage Classifier

DistilBERT finetuned on 10,980 synthetic property maintenance tickets with two classification heads:

• Category (10 classes): Appliances, Common Areas, Electrical, Elevator, HVAC, Noise, Pests, Plumbing, Security, Structural
• Urgency (4 classes): P1 Emergency (4h), P2 Urgent (24h), P3 Standard (3-5d), P4 Scheduled (7-14d)

Part of the PropertyPilot project — an AI-powered property maintenance triage system.

Performance

Task	Zero-shot F1	Finetuned F1	Improvement
Category (macro)	0.032	0.972	+0.939
Urgency (macro)	0.062	0.761	+0.700

Urgency labels are derived from ticket text content (keyword-based rule classifier), making them strongly correlated with the actual maintenance scenario described.

Dataset

Trained on propertypilot/property-pilot-tickets:

• 13,726 clean rows, 10 categories, 4 urgency tiers
• Brackets/template placeholders stripped, ispers corruption fixed
• 80/10/10 stratified train/val/test split

Model Architecture

DistilBertModel (distilbert-base-uncased)
  -> CLS token -> Dropout(0.1)
  -> Linear(768, 10)  # category head
  -> Linear(768, 4)   # urgency head

Training

• Epochs: 5, LR: 2e-5, Batch: 32, Max seq len: 128
• AdamW + linear warmup (10%) + gradient clipping
• Joint loss: CrossEntropy(category) + CrossEntropy(urgency)

Files

File	Description
model.pt	PyTorch state dict (best checkpoint by combined val F1)
label_map.json	id2label / label2id for both heads
tokenizer_config.json	Tokenizer config (distilbert-base-uncased fast tokenizer)
tokenizer.json	Fast tokenizer file
distilbert_config.json	Base DistilBERT config
eval_results.json	Full training history + test metrics
confusion_cat.png	Category confusion matrix
confusion_urg.png	Urgency confusion matrix

Inference

import torch, json
from transformers import DistilBertModel, DistilBertTokenizerFast
import torch.nn as nn

class TriageClassifier(nn.Module):
    def __init__(self, n_cat, n_urg):
        super().__init__()
        self.bert = DistilBertModel.from_pretrained("distilbert-base-uncased")
        self.dropout = nn.Dropout(0.1)
        self.cat_head = nn.Linear(768, n_cat)
        self.urg_head = nn.Linear(768, n_urg)

    def forward(self, input_ids, attention_mask):
        out = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        cls = self.dropout(out.last_hidden_state[:, 0])
        return self.cat_head(cls), self.urg_head(cls)

# Load
label_map = json.load(open("label_map.json"))
tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-uncased")
model = TriageClassifier(n_cat=10, n_urg=4)
model.load_state_dict(torch.load("model.pt", map_location="cpu"))
model.eval()

# Predict
text = "gas leak in kitchen, can smell it strongly"
enc = tokenizer(text, return_tensors="pt", max_length=128, truncation=True, padding="max_length")
with torch.no_grad():
    cat_logits, urg_logits = model(enc["input_ids"], enc["attention_mask"])

cat = label_map["category_id2label"][str(cat_logits.argmax(-1).item())]
urg = label_map["urgency_id2label"][str(urg_logits.argmax(-1).item())]
print(f"Category: {cat} | Urgency: {urg}")