| --- |
| |
| |
| {} |
| --- |
| |
| # Model Card for Model ID |
|
|
| <!-- Provide a quick summary of what the model is/does. --> |
|
|
| This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/netgvarun2012/VirtualTherapist). |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| <!-- Provide a longer summary of what this model is. --> |
| A MultiModal architecture model that was created and finetuned jointly by concatenating Hubert and BERT embeddings. |
| Hubert model was fine-tuned with a classification head on preprocessed audio and emotion labels in supervised manner. |
| BERT was trained on text transcrition embeddings. |
|
|
| Model can accurately recognize emotions classes- Angry,Sad,Fearful,Happy,Disgusted,Surprised,Calm with ~80% accuracy. |
|
|
|
|
| - **Developed by:** [https://www.linkedin.com/in/sharmavaruncs/] |
| - **Model type:** [MultiModal - Text and Audio based] |
| - **Language(s) (NLP):** [NLP, Speech processing] |
| - **Finetuned from model [optional]:** [https://huggingface.co/docs/transformers/model_doc/hubert] |
| |
| ### Model Sources [optional] |
| |
| <!-- Provide the basic links for the model. --> |
| |
| - **Repository:** [https://github.com/netgvarun2012/VirtualTherapist/] |
| - **Paper [optional]:** [https://github.com/netgvarun2012/VirtualTherapist/blob/main/documentation/Speech_and_Text_based_MultiModal_Emotion_Recognizer.pdf] |
| - **Demo [optional]:** [https://huggingface.co/spaces/netgvarun2005/VirtualTherapist] |
| |
| ## Uses |
| |
| <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
| 'Virtual Therapist' app - an Intelligent speech and text input based assistant that can decipher emotions and generate therapeutic messages based on the Emotional state of the user. |
| |
| Emotions recognized - Angry,Sad,Fearful,Happy,Disgusted,Surprised,Calm with ~80% accuracy. |
| |
| Use the code below to get started with the model: |
| |
| |
| class MultimodalModel(nn.Module): |
| ''' |
| Custom PyTorch model that takes as input both the audio features and the text embeddings, and concatenates the last hidden states from the Hubert and BERT models. |
| ''' |
| def __init__(self, bert_model_name, num_labels): |
| super().__init__() |
| self.hubert = HubertForSequenceClassification.from_pretrained("netgvarun2005/HubertStandaloneEmoDetector", num_labels=num_labels).hubert |
| self.bert = AutoModel.from_pretrained(bert_model_name) |
| self.classifier = nn.Linear(self.hubert.config.hidden_size + self.bert.config.hidden_size, num_labels) |
| |
| def forward(self, input_values, text): |
| hubert_output = self.hubert(input_values).last_hidden_state |
| |
| bert_output = self.bert(text).last_hidden_state |
| |
| # Apply mean pooling along the sequence dimension |
| hubert_output = hubert_output.mean(dim=1) |
| bert_output = bert_output.mean(dim=1) |
| |
| concat_output = torch.cat((hubert_output, bert_output), dim=-1) |
| logits = self.classifier(concat_output) |
| return logits |
| |
|
|
| def load_model(): |
| """ |
| Load and configure various models and tokenizers for a multi-modal application. |
| |
| This function loads a multi-modal model and its weights from a specified source, |
| initializes tokenizers for the model and an additional language model, and returns |
| these components for use in a multi-modal application. |
| |
| Returns: |
| tuple: A tuple containing the following components: |
| - multiModel (MultimodalModel): The multi-modal model. |
| - tokenizer (AutoTokenizer): Tokenizer for the multi-modal model. |
| - model_gpt (AutoModelForCausalLM): Language model for text generation. |
| - tokenizer_gpt (AutoTokenizer): Tokenizer for the language model. |
| """ |
| # Load the model |
| multiModel = MultimodalModel(bert_model_name, num_labels) |
| |
| # Load the model weights and tokenizer directly from Hugging Face Spaces |
| multiModel.load_state_dict(torch.hub.load_state_dict_from_url(model_weights_path, map_location=device), strict=False) |
| tokenizer = AutoTokenizer.from_pretrained("netgvarun2005/MultiModalBertHubertTokenizer") |
| |
| # GenAI |
| tokenizer_gpt = AutoTokenizer.from_pretrained("netgvarun2005/GPTTherapistDeepSpeedTokenizer", pad_token='<|pad|>',bos_token='<|startoftext|>',eos_token='<|endoftext|>') |
| model_gpt = AutoModelForCausalLM.from_pretrained("netgvarun2005/GPTTherapistDeepSpeedModel") |
| |
| return multiModel,tokenizer,model_gpt,tokenizer_gpt |
| |
|
|
|
|
|
|
| ## Model Card Authors [Varun Sharma] |
|
|
| |
|
|
