Each image have:
Labels from json file
[
{
"id": 1,
"file_path": "Part1/1/0.jpg",
"attributes": [
"woman,short hair,black jacket,blue denim jeans,black sneakers,black backpack"
],
"captions": [
"A woman with black hair and she is wearing a black jacket with blue jeans paired with black shoes."
],
"prompt_caption": [
"The woman has short hair. She is wearing a black jacket, blue denim jeans and black sneakers. She is carrying a black backpack. "
]
},
{
"id": 1,
"file_path": "Part1/1/1.jpg",
"attributes": [
"girl,dark hair,black jacket,blue jeans,black shoes,unknown"
],
"captions": [
"A woman wearing a black jacket over a Gray shirt, a pair of blue jeans and a pair of black shoes."
],
"prompt_caption": [
"A girl with dark hair wearing a black jacket and blue jeans with black shoes."
]
}, ...]
Step 1: Manual labeling
Step 2: Train model SPAC to generate a texual caption and list of 6 attribute keywords for image
Step 3: Attributes Prompt Engineering
Data flow for generating different captions for each image
Flow for generating attributes phrases (keywords)
Final loss for SPAC $$\mathcal{L_{generate}} = \lambda \mathcal{L}_{attribute} + \mathcal{L}_{caption} $$
Vision Encoder
Text Encoder
Additional module for learning pretext task:
Vision-Matching Matching (VLM): just a Constrastive Learning module
Visual-fused Attributes Prediction (VAP): module learn masked text prediction from image clues
Semantic-fused Image Colorization (SIC): module learn image colorization from text clues
Just other name of vision-language constrastive learning
Global visual feature vector
Global text feature vector
Recall the CMPM loss
From paper "Deep Cross-Modal Projection Learning for Image-Text Matching", ECCV 2018
Given a mini-batch with $n$ image and text samples, for each image $\boldsymbol{x}_i$ the image-text pairs are constructed as $\left\{\left(\boldsymbol{x}_i, \boldsymbol{z}_j\right), y_{i, j}\right\}_{j=1}^n$, where $y_{i, j}=1$ means that $\left(\boldsymbol{x}_i, \boldsymbol{z}_j\right)$ is a matched pair, while $y_{i, j}=0$ indicates the unmatched ones. The probability of matching $\boldsymbol{x}_i$ to $\boldsymbol{z}_j$ is defined as: $$ p_{i, j}=\frac{\exp \left(\boldsymbol{x}_i^{\top} \overline{\boldsymbol{z}}_j\right)}{\sum_{k=1}^n \exp \left(\boldsymbol{x}_i^{\top} \overline{\boldsymbol{z}}_k\right)} \quad \text { s.t. } \overline{\boldsymbol{z}}_j=\frac{\boldsymbol{z}_j}{\left\|\boldsymbol{z}_j\right\|} $$
Normalized Ground-true vector: $$q_{i, j}=\frac{y_{i, j}}{\sum_{k=1}^n y_{i, k}}$$
The matching loss of associating $\boldsymbol{x}_i$ with correctly matched text samples is defined as $$ \mathcal{L}_i= KL(\mathbf{p_i}|| \mathbf{q_i})= \sum_{j=1}^n p_{i, j} \log \frac{p_{i, j}}{q_{i, j}+\epsilon} $$ where $\epsilon$ is a small number to avoid numerical problems, and the matching loss from image to text in a mini-batch is computed by $$ \mathcal{L}_{i 2 t}=\frac{1}{n} \sum_{i=1}^n \mathcal{L}_i $$
$$\mathcal{L}_{CMPM} = \mathcal{L}_{i2t} + \mathcal{L}_{t2i}$$
PLIP apply CMPM loss for VLM
$$\mathcal{L}_{VLM} = \mathbb{E}_{\sim minibatch}[\mathcal{L}_{CMPM}(v_{g}, t_{g})]$$
Loss for pretraining is not shown in source code! Just vision encoder and text encoder are shown
import os, sys
print(os.getcwd())
from pathlib import Path
ROOT_PATH = Path('../paper_clones/PLIP').resolve(); sys.path.append(str(ROOT_PATH))
FINETUNE_PATH = ROOT_PATH/'Downstreams/CMPM-C'; sys.path.append(str(FINETUNE_PATH))
IMAGE_PATH = Path('../DATASET').resolve(); sys.path.append(str(IMAGE_PATH))
import torchvision.transforms as transforms
from torchinfo import summary
import torch
import yaml
from utils import *
import time
import os
import shutil
import torch.backends.cudnn as cudnn
from test_dataloader import get_loader_test
import pickle
import argparse
from PLIPmodel import Create_PLIP_Model
class MyArgumentParser:
def __init__(self):
self.args = {}
def __str__(self):
return str(self.args)
def add_argument(self, str_arg, type, default, help=None):
self.args[str_arg[2:]] = type(default)
setattr(self, str_arg[2:], type(default))
def parse_args(self):
return self
def test_parse_args(parser):
parser.add_argument('--model_path', type=str, default='checkpoints/PLIP_MRN50.pth.tar')
parser.add_argument('--image_path', type=str, default='data/CUHK-PEDES/imgs')
parser.add_argument('--test_path', type=str,
default='data/CUHK-PEDES/CUHK-PEDES-test.json',
help='path for test annotation json file')
# ***********************************************************************************************************************
# 设置模型backbone的类型和参数
parser.add_argument('--plip_model', type=str, default='MResNet_BERT')
parser.add_argument('--img_backbone', type=str, default='ModifiedResNet',
help="ResNet:xxx, ModifiedResNet, ViT:xxx")
parser.add_argument('--txt_backbone', type=str, default="bert-base-uncased")
parser.add_argument('--img_dim', type=int, default=768, help='dimension of image embedding vectors')
parser.add_argument('--text_dim', type=int, default=768, help='dimension of text embedding vectors')
parser.add_argument('--layers', type=list, default=[3, 4, 6, 3], help='Just for ModifiedResNet model')
parser.add_argument('--heads', type=int, default=8, help='Just for ModifiedResNet model')
parser.add_argument('--height', type=int, default=256)
parser.add_argument('--width', type=int, default=128)
# 设置超参数
parser.add_argument('--batch_size', type=int, default=128)
parser.add_argument('--num_workers', type=int, default=4)
parser.add_argument('--device',type=str,default="cuda:0")
parser.add_argument('--feature_size', type=int, default=768)
args = parser.parse_args()
return args
args = test_parse_args(MyArgumentParser())
model = Create_PLIP_Model(args).to('cpu')
model_file = args.model_path
modules = [x for x in dict(model.named_modules()).keys() if '.' not in x and len(x) > 0]
modules
Text encoder
summary(model.text_encoder)
modules = [x for x in dict(model.text_encoder.encoder.named_modules()).keys() if '.' not in x and len(x) > 0]
modules
model.text_encoder.encoder.embeddings
summary(model.text_encoder.encoder.encoder)
model.text_encoder.encoder.pooler
Image Encoder
summary(model.image_encoder)