project-creation-quality-distilbert-v1

Version 1 of the MyVillage V3 project-creation quality classifier. It is DistilBERT (distilbert/distilbert-base-uncased, ~67M parameters), fine-tuned as a 12-label multilabel classifier over the semantic and hybrid failure codes of taxonomy v3.0.0.

It is meant to run inside the hybrid system. The bundled v3cls/ package adds the deterministic validator (7 rule-based codes and the hybrid rules) and the decision engine. The system returns exactly:

{"decision": "ACCEPT" | "REJECT", "failed_requirements": ["CODE", ...]}

Usage

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("mjpsm/project-creation-quality-distilbert-v1")
sys.path.insert(0, path)
from v3cls.inference import V3Classifier
clf = V3Classifier(path)
clf.classify(prompt="...", title="...", description="...")

Data

There are 1,000 synthetic and golden-derived records (300 accept / 490 single-violation / 210 multi-violation), split 800/100/100 by concept. Thresholds were calibrated on validation (v3_thresholds.json). All metrics below are on the untouched test split.

Test results (full hybrid system)

metric value
decision accuracy 0.94
reject precision / recall / F1 1.0 / 0.9143 / 0.9552
false accepts / false rejects 6 / 0
exact match of failed_requirements 0.77

Test results (model labels, calibrated thresholds)

label detector threshold support precision recall F1
THEME_MISALIGNMENT semantic 0.5 8 0.636 0.875 0.737
AGE_INAPPROPRIATE semantic 0.22 8 0.875 0.875 0.875
UNBOUNDED_DELIVERABLE semantic 0.24 2 0.286 1.0 0.444
NO_TECH_ARTIFACT semantic 0.5 8 0.889 1.0 0.941
WEAK_DEFINITION_OF_DONE semantic 0.5 8 0.875 0.875 0.875
PRESCRIBED_TOOLING hybrid 0.5 7 1.0 0.857 0.923
PROCEDURAL_INSTRUCTIONS hybrid 0.5 7 1.0 1.0 1.0
COLLABORATION_NOT_INTEGRAL semantic 0.5 8 1.0 1.0 1.0
PARAGRAPH_ROLE_VIOLATION semantic 0.52 9 1.0 0.778 0.875
SAFETY_OR_OVERCLAIM semantic 0.38 8 1.0 0.125 0.222
TITLE_QUALITY semantic 0.62 6 0.5 0.5 0.5
LEARNING_OUTCOME_MISSING semantic 0.5 4 1.0 1.0 1.0

Limitations

The test split is small: 100 records, with 2–9 positives per label. Thresholds come from a 100-record validation split. The training data is mostly synthetic contrastive edits of compliant bases. Review the outputs before relying on this model for production gating.

Downloads last month
37
Safetensors
Model size
67M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for mjpsm/project-creation-quality-distilbert-v1

Finetuned
(12,886)
this model

Space using mjpsm/project-creation-quality-distilbert-v1 1