Mirror of Khriis/PAIR

README.md

@@ -0,0 +1,129 @@
+---
+library_name: transformers
+pipeline_tag: text-classification
+language:
+  - en
+tags:
+  - reflection
+  - cross-encoder
+  - motivational-interviewing
+  - conversation-analysis
+  - roberta
+---
+
+# PAIR Reflection Scorer (Cross‑Encoder)
+
+This repository provides weights for a PAIR‑style cross‑encoder that scores the quality of counselor reflections in Motivational Interviewing (MI). Given a client/patient prompt and a counselor response, the model outputs a scalar score in [0,1] indicating how strongly the response reflects the prompt.
+
+This model is based on the approach described in:
+
+- Min, Do June; Pérez‑Rosas, Verónica; Resnicow, Kenneth; Mihalcea, Rada. “PAIR: Prompt‑Aware margIn Ranking for Counselor Reflection Scoring in Motivational Interviewing.” EMNLP 2022. https://aclanthology.org/2022.emnlp-main.11/
+
+Please credit the authors above when using this model or derivative work.
+
+## Task & Motivation (from the paper)
+
+- Reflections are a core verbal counseling skill used to convey understanding and acknowledgment of clients’ experiences.
+- The goal is to automatically score counselor reflections to provide timely, useful feedback for training and education.
+- Input to the scorer: a dialog turn consisting of a client prompt (likely to elicit a reflection) and the counselor’s response.
+- Output: a numeric reflection score capturing the quality/strength of the reflection.
+
+## Method: Prompt‑Aware Margin Ranking (PAIR)
+
+PAIR trains a prompt‑aware cross‑encoder that contrasts positive and negative (prompt, response) pairs. The key idea is to learn, for a given prompt, to rank higher‑quality reflections above lower‑quality or mismatched responses using margin‑based ranking losses.
+
+High‑level components reflected by this implementation:
+
+- Encoder: `roberta-base` cross‑encoder over concatenated (prompt, response).
+- Scoring head: Small MLP over the [CLS] token (768 → 512 → 1) with ELU.
+- Training objective (as per the paper/code): multi‑gap margin ranking that separates:
+  - High‑quality (HQ) reflections from medium‑quality (MQ) and low‑quality (LQ).
+  - HQ/MQ reflections from explicit mismatches (responses paired with the wrong prompt).
+- Inference: apply sigmoid to the logit to obtain a reflection score in [0,1].
+
+The included `cross_scorer_model.py` shows the MLP head and margin losses consistent with a PAIR‑style training setup.
+
+## Files
+
+- `reflection_scorer_weight.pt` — fine‑tuned cross‑encoder weights (encoder + head).
+- `cross_scorer_model.py` — `CrossScorerCrossEncoder` module used for inference/training.
+- `min_pair_2022.txt` — text version summary of the PAIR paper (for reference in this repo).
+
+## Intended Use & Limitations
+
+- Intended for research, education, and tooling around reflection scoring in counseling‑style conversations.
+- Not a clinical or diagnostic tool; do not use for high‑stakes decisions.
+- Scores are not calibrated probabilities; treat relative differences with caution.
+- As with all ML models, outputs may reflect biases in pretraining/fine‑tuning data.
+
+## Quickstart
+
+```python
+from huggingface_hub import hf_hub_download
+from transformers import AutoModel, AutoTokenizer
+import torch, importlib.util, sys
+
+repo_id = "Khriis/PAIR"  # replace if you fork
+
+# 1) Download weights and model code from the repo
+ckpt_path = hf_hub_download(repo_id=repo_id, filename="reflection_scorer_weight.pt")
+code_path = hf_hub_download(repo_id=repo_id, filename="cross_scorer_model.py")
+
+# 2) Import model definition
+spec = importlib.util.spec_from_file_location("cross_scorer_model", code_path)
+mod = importlib.util.module_from_spec(spec)
+sys.modules["cross_scorer_model"] = mod
+spec.loader.exec_module(mod)
+
+# 3) Build encoder + head and load state dict
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+encoder = AutoModel.from_pretrained("roberta-base", add_pooling_layer=False)
+model = mod.CrossScorerCrossEncoder(encoder).to(device)
+tokenizer = AutoTokenizer.from_pretrained("roberta-base")
+
+state = torch.load(ckpt_path, map_location=device)
+sd = state.get("model_state_dict", state)
+model.load_state_dict(sd)
+model.eval()
+
+# 4) Score a (prompt, response) pair
+prompt = "I’ve been overwhelmed at work and can’t focus."
+response = "It sounds like you’re under a lot of pressure, and it’s affecting your ability to concentrate."
+batch = tokenizer(prompt, response, padding="longest", truncation=True, return_tensors="pt").to(device)
+with torch.no_grad():
+    score = model.score_forward(**batch).sigmoid().item()
+print("Reflection score:", round(score, 3))
+```
+
+### Using in the Toolkit
+
+The toolkit can download the file automatically (public repo). For offline use, place `reflection_scorer_weight.pt` locally and set `REFLECTION_CKPT_PATH` to that path.
+
+## Citation
+
+If you use this model or code, please cite the PAIR paper:
+
+Informal citation: “PAIR: Prompt‑Aware margIn Ranking for Counselor Reflection Scoring in Motivational Interviewing” (Min et al., EMNLP 2022). https://aclanthology.org/2022.emnlp-main.11/
+
+BibTeX (adapt based on official entry):
+
+```bibtex
+@inproceedings{min-etal-2022-pair,
+  title     = {PAIR: Prompt-Aware margIn Ranking for Counselor Reflection Scoring in Motivational Interviewing},
+  author    = {Min, Do June and P{\'e}rez-Rosas, Ver{\'o}nica and Resnicow, Kenneth and Mihalcea, Rada},
+  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
+  year      = {2022},
+  url       = {https://aclanthology.org/2022.emnlp-main.11/}
+}
+```
+
+Also cite RoBERTa:
+
+```bibtex
+@misc{liu2019roberta,
+  title         = {{RoBERTa}: A Robustly Optimized {BERT} Pretraining Approach},
+  author        = {Liu, Yinhan and others},
+  year          = {2019},
+  url           = {https://arxiv.org/abs/1907.11692}
+}
+```

config.json

@@ -0,0 +1,4 @@
+{
+  "task": "text-classification",
+  "custom_handler": "handler.py"
+}

cross_scorer_model.py

@@ -0,0 +1,160 @@
+import transformers
+import torch.nn as nn
+from torch.nn.utils.rnn import pad_sequence
+from transformers import BertTokenizer, BertModel
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+from transformers.modeling_outputs import MaskedLMOutput, SequenceClassifierOutput
+from transformers import BertForMaskedLM
+
+import torch.nn.functional as F
+
+import transformers
+import torch.nn as nn
+
+
+class CrossScorerCrossEncoder(nn.Module):
+
+    def __init__(self, transformer):
+
+        super(CrossScorerCrossEncoder, self).__init__()
+
+        self.cross_encoder = transformer
+        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+        
+
+        # Binary Head
+        self.l1 = torch.nn.Linear(768, 512)
+        self.relu = torch.nn.ELU()
+        self.l2 = torch.nn.Linear(512,1)
+
+        self.encoder_type = "cross"    
+
+
+    def score_forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        token_type_ids=None,
+        position_ids=None,
+        head_mask=None,
+        inputs_embeds=None,
+        encoder_hidden_states=None,
+        encoder_attention_mask=None,
+        labels=None,
+        output_attentions=None,
+        output_hidden_states=None,
+        return_dict=None,
+        return_attentions=False
+        ):  
+
+
+        output = self.cross_encoder(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            encoder_hidden_states=encoder_hidden_states,
+            encoder_attention_mask=encoder_attention_mask,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        pair_reps = output.last_hidden_state[:,0,:]
+        score = self.l2(self.relu(self.l1(pair_reps)))
+        
+        if output_attentions and return_attentions:
+            return score.sigmoid().squeeze(), output.attentions
+
+        return score
+
+
+    def cl_loss(self, pair_scores, labels):
+        BSZ = pair_scores.size(0) 
+        BSZ = int(BSZ/(4))
+
+        pair_scores= list(pair_scores.tensor_split(BSZ, dim=0) )
+        pair_scores = torch.stack(pair_scores)
+        
+        
+        gap_1_loss_fct = nn.MarginRankingLoss(margin=0.5)
+        gap_2_loss_fct = nn.MarginRankingLoss(margin=1.0)
+              
+        mq_scores = pair_scores[:,1] # 1
+        lq_scores = pair_scores[:,2:-1] # 2
+
+
+        hq_scores = pair_scores[:,0] 
+
+        hq_mq_loss = gap_1_loss_fct(
+                hq_scores.flatten(), 
+                mq_scores.flatten(), 
+                torch.ones(mq_scores.flatten().size()).to(self.device))
+        mq_lq_loss = gap_1_loss_fct(
+                mq_scores.repeat(1,lq_scores.size(-1)).flatten(), 
+                lq_scores.flatten(), 
+                torch.ones(lq_scores.flatten().size()).to(self.device))
+        hq_lq_loss = gap_2_loss_fct(
+                hq_scores.repeat(1,lq_scores.size(-1)).flatten(), 
+                lq_scores.flatten(), 
+                torch.ones(lq_scores.flatten().size()).to(self.device))
+        
+        mismatch_scores = pair_scores[:,-1]
+        hq_mismatch_loss =  gap_2_loss_fct(
+                        hq_scores.flatten(), 
+                        mismatch_scores.flatten(), 
+                        torch.ones(mismatch_scores.flatten().size()).to(self.device))
+        mq_mismatch_loss = gap_1_loss_fct(
+                mq_scores.flatten(), 
+                mismatch_scores.flatten(), 
+                torch.ones(mismatch_scores.flatten().size()).to(self.device))
+        mismatch_loss = hq_mismatch_loss + mq_mismatch_loss 
+
+        loss = hq_mq_loss + mq_lq_loss + hq_lq_loss  + mismatch_loss 
+        return loss
+
+    def forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        token_type_ids=None,
+        position_ids=None,
+        head_mask=None,
+        inputs_embeds=None,
+        encoder_hidden_states=None,
+        encoder_attention_mask=None,
+        labels=None,
+        output_attentions=None,
+        output_hidden_states=None,
+        return_dict=None,
+        random = False
+        ):
+ 
+        pair_scores = self.score_forward(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            encoder_hidden_states=encoder_hidden_states,
+            encoder_attention_mask=encoder_attention_mask,
+            labels=labels,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        ).sigmoid().squeeze()
+
+
+            
+        cl_loss = self.cl_loss(pair_scores, labels)
+
+        loss =   cl_loss 
+        return SequenceClassifierOutput(
+                loss=loss,
+                logits=pair_scores,
+                )
+ 

handler.py

@@ -0,0 +1,114 @@
+import torch
+import importlib.util
+import sys
+import os
+from typing import Dict, List, Any
+from transformers import AutoModel, AutoTokenizer
+
+class EndpointHandler():
+    def __init__(self, path=""):
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        
+        # Import custom model definition from local file
+        # The file is expected to be in the same directory as handler.py, which 'path' points to
+        model_filename = "cross_scorer_model.py"
+        model_path = os.path.join(path, model_filename) 
+        
+        # Fallback if path is empty or "." and file is in CWD
+        if not os.path.exists(model_path):
+             model_path = model_filename
+
+        spec = importlib.util.spec_from_file_location("cross_scorer_model", model_path)
+        mod = importlib.util.module_from_spec(spec)
+        sys.modules["cross_scorer_model"] = mod
+        spec.loader.exec_module(mod)
+        
+        # Initialize encoder and custom model
+        encoder = AutoModel.from_pretrained("roberta-base", add_pooling_layer=False)
+        self.model = mod.CrossScorerCrossEncoder(encoder).to(self.device)
+        
+        # Load weights
+        weights_filename = "reflection_scorer_weight.pt"
+        weights_path = os.path.join(path, weights_filename)
+        
+        if not os.path.exists(weights_path):
+            weights_path = weights_filename
+
+        state = torch.load(weights_path, map_location=self.device)
+        sd = state.get("model_state_dict", state)
+        self.model.load_state_dict(sd, strict=False)
+
+        load_res = self.model.load_state_dict(sd, strict=False)
+        
+        missing = load_res.missing_keys
+        unexpected = load_res.unexpected_keys
+        
+        # print(f"[PAIR] weights_path={weights_path}")
+        # print(f"[PAIR] loaded_keys={len(sd.keys())} missing_keys={len(missing)} unexpected_keys={len(unexpected)}")
+        # print(f"[PAIR] missing_keys_sample={missing[:20]}")
+        # print(f"[PAIR] unexpected_keys_sample={unexpected[:20]}")
+        
+        self.model.eval()
+        
+        # Initialize tokenizer
+        self.tokenizer = AutoTokenizer.from_pretrained("roberta-base")
+
+    def __call__(self, data: Dict[str, Any]) -> List[Dict[str, Any]]:
+        """
+        data args:
+            inputs (:obj: `list` | `dict`): The inputs to the model.
+        """
+        # get inputs
+        inputs = data.pop("inputs", data)
+        
+        # If inputs is a dict (single item), wrap in list to reuse logic, or handle list
+        if isinstance(inputs, dict):
+            inputs = [inputs]
+        
+        results = []
+        for item in inputs:
+            prompt = item.get("prompt")
+            response = item.get("response")
+            
+            if not prompt or not response:
+                results.append({"error": "Missing prompt or response"})
+                continue
+
+            # Preprocessing
+            prompt_fmt = f"Client: {prompt}"
+            response_fmt = f"Therapist: {response}"
+            batch = self.tokenizer(prompt_fmt, response_fmt, padding="longest", truncation=True, return_tensors="pt").to(self.device)
+                        
+            # Inference
+            with torch.no_grad():
+                # score_forward returns raw logits (based on README/code usage), we need sigmoid
+                score = self.model.score_forward(**batch).sigmoid().item()
+
+            # Preprocessing (A: prompt->response)
+            batch_a = self.tokenizer(
+                prompt, response,
+                padding="longest", truncation=True, return_tensors="pt"
+            ).to(self.device)
+            
+            # Preprocessing (B: response->prompt)
+            batch_b = self.tokenizer(
+                response, prompt,
+                padding="longest", truncation=True, return_tensors="pt"
+            ).to(self.device)
+            
+            with torch.no_grad():
+                logit_a = self.model.score_forward(**batch_a).item()
+                prob_a  = torch.sigmoid(torch.tensor(logit_a)).item()
+            
+                logit_b = self.model.score_forward(**batch_b).item()
+                prob_b  = torch.sigmoid(torch.tensor(logit_b)).item()
+            
+            # print(f"[PAIR] A(prompt->resp) logit={logit_a:.6f} prob={prob_a:.6f}")
+            # print(f"[PAIR] B(resp->prompt) logit={logit_b:.6f} prob={prob_b:.6f}")
+            
+            # TEMP: return the higher one to validate quickly
+            score = max(prob_a, prob_b)
+            
+            results.append({"score": score})
+            
+        return results

reflection_scorer_weight.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81eb3a3a9240b9bc05b7ca79b1e7920045917504965c7233fbc32e0c3b99ebf5
+size 497885869

requirements.txt

@@ -0,0 +1,2 @@
+transformers
+torch