Paper Information¶

• Paper title: Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark
• Date Published: 14/8/2023
• Conference: ACM Multi-media 2023
• Link github: https://github.com/Shuyu-XJTU/APTM
• Link paper: https://arxiv.org/abs/2306.02898
• Why research this paper?
  • SOTA method
  • New paradism for learning Pestrian Attribute Recognition (PAR)
  • Join learning PAR and Text-based Person Retrieval (TPBR) in a joint framework
  • Prosed a new method to generate large-scale dataset for TBPR
  • Construct a 1.5M (image-captioning) dataset for human-centric pretraining

Overview/Main contribution¶

Main contribution¶

• Create a new large-scale dataset for pretraining, contained:
  • 1.5M images
  • Attribute labels for each image: one-hot encoding format
  • Text captioning for each image, focus on attribute description
  • Interesting fact: only 0.3/1.5M images is enough to get SOTA benchmark, more is unnessary
  
  
• Propose new method Attribute Prompt Learning (APL) for attibute recognition:
  • Replace for multilabel classification approach that based on cross entropy loss
  • Each attribute label is convert to a short text prompt (ex: label "man: -> prompt "This person is a man")
  
  
• Join APL and Text Matching learning (TM) in a framework for pretraining:
  • TM: learn the relation between text and image, AP + TM -> APTM
  • An end-to-end multitask learning framework
  • Backbone after pretraining can be downstream for two task:
    • Text-based Person Retrieval (TBPR)
    • Pedestrian Attribute Recognition (PAR)
  
  
• For each of APL and ITM, there are 3 objective function for 3 pretext task (consider short attribute prompt as short text caption):
  • Masked Language Modeling (MLM):
    • Given an image and masked-caption, try to predict what the original word at masked position
      • Like the generation task for language model, but at masked position
    • A pretext task of pretraning model BERT, the name is used to differentiate with Casual Language Modeling task belong to lanugage model
      • Masked Language Modeling: bi-directional generating
      • Casual Language Modeling: left-to-right and autoregressive generating
  • Image-Text Contrastive Learning:
    • Try to make the vectors belong to a similar text-image pair to be nearer in common crossmodal space, while pull away the dissimilar pairs
    • An important loss, that make the inference phase be simple just by calculate the cosine similariy
  • Image-Text Binary Matching:
    • Given a pair of image and text-caption, try to predict this pair is match or unmatched -> binary classification
    • The common way to filter and re-ranking in inference phase: after getting the candidate list from searching by cosine similarity
  
  
• In summary, there are total 6 loss for pretraining:
  • The final loss for pretraining is weighted sum of 6 loss
  • For finetune on PAR dataset:
    • Only 3 loss is used (for image - attribute prompt) pairs
  • For finetune on TPBR dataset: for each image, sample an ground-true text caption, and an augmentation for this text caption --> 1 image correspond to 2 text
    • 3 loss for original text caption
    • 3 loss for augmentation text caption
    • Final loss is weighted sum of these 6 loss

Overview dataset¶

General pretraining dataset - MALS¶

How dataset was created?¶

Text caption label¶

• Step 1: Given a pair of text caption - image, use diffusion model ImaginAIry to collect the synthetic images:
  • Link model: https://github.com/brycedrennan/imaginAIry
  • The images is so noisy, that need to be post-processed
  
  
• Step 2: Pose-process after generating synthetic images to remove noisy image:
  • Sort images by file size and delete images whose size is smaller than 24kb to filter out blurred images.
  • Compute the mean variance of the difference between the 3 channels of every image and remove images whose mean variance is less than a presetting threshold.
  • Apply OpenPose model to detect human key points and filter out the undesired person images.
  • Leverage the detected key points as a tight bounding box to re-crop the samples.
  
  
• Step 3: Use BLIP model to produce more fitting captions for every synthetic image
  • Use each new caption for each synthetic image as a final text-image pair

Attribute Label¶

• Define the attribute space in the same way as Market-1501 Attribute dataset, which have 27 attribute
  
• Use two mechanisms to obtain attributes from the text captiion: Explicit Matching (EM) and Implicit Extension (IE)
  • Explicit Matching:
    • Infer the correspondence of specific attributes based on keywords in the text, such as the keyword "man" in text caption will correspond to the attribute "gender: male"
  • Implicit Extension:
    • Assign corresponding attribute candidates based on features that are not mentioned in the text
    • Such as samples that don't mention "hat" in their descriptions, will be labeled "hat: no"

Dataset properties¶

• Large-scale: 1.5M images

• Diversity and High-fidelity Images:
  • Diverse of human 's attributes
  • But the face for the people is low quality, not real !

Label Format¶

From json file

• What label means?
  • 0: Fasle
  • 1: True
  • -1: Unknown/Can't be inferred

Attribute prompts correspond to attribute label

Downstream dataset¶

• ICFG-PEDES
• CUHK-PEDES
• RSTPReID

Model Architecture / Method¶

Architecture¶

Image from paper

Re-drawed Image

Link to whiteboard: just register email, then view: https://miro.com/welcomeonboard/VktCZkRHcjJIZ2NzVmtjaHVNcHdOMXpORThsZE1OSk9aNEJVa2FBRWpFa1pSQTd0Wnk4RzRrbG93S29hZFVIWXwzMDc0NDU3MzYzMzk5NTU1NzQ1fDI=?share_link_id=935922303615

Main component¶

• Typical architecture to deal with crossmodal information:
  • A vision encoder (transformer-based) to encode the feature of image:
    • Swin Transformer base with ~80M pretraining parameters
    • Encode an image into sequence of patch embedding
      • Add CLS token to the start of sequence , use as the global represenatation for image
      • Patch embedding use as the local representation for image
      • Whole sequence will be fed to cross-attention module of cross encoder as key and value
    • Output dimension: 1024
      • The CLS token embedding will be linear projected into 256-d space to use as image 's global feature vector.
  • A language encoder based on transformer to encode the feature of caption text / attribute short prompt:
    • First 6 encoder layers of pretraining model BERT base (original BERT has 12)
    • Encode the whole text into the sequence of token embedding:
      • Each token embedding the sum of word embedding, segment (sentence) embedding and learnable postition embedding (all embeddings are learable)
      • Special token:
        • [CLS] token as the global represenation of whole text
        • [SEP] token as the separator between sentences
        • [PAD] token as the embedding for padded position (each sentence is padding to the same length to convinient for batch-calculating)
      • Output dimension: 768
        • The CLS embedding vector will be linear projected into 256-d space to use as text 's global feature vector
  • A cross encoder:
    • Behave as transformer decoder:
      • But use bi-directional generating mechanism
      • Original transformer decoder generate in left-to-right casual and autoregressive manner
    • Use last 6 encoder layers of model BERT as cross encoder
    • Add cross-attention module after the self-attention module in each layer
    • Use text representations (1024-d) as query, use visual representation as key and value (768-d)
    • Output dimension: 256-d
      • Use output CLS embedding as the global representation vector for a pair of text - image -> feed to a MLP to check a pair matched or unmatched

• For Masked Language Modeling (MLM) task:
  • Need a MLP for predicted the word as the masked position
  • For each image and a positive masked-text caption:
    • Use output embedding vectors at masked positions to feed in the MLP for prediction

• For Image-Text Matching task:
  • Need a MLP to binarry classify, which receive the global representation vector (CLS embedding) of a pair (image-text)

• In summary:
  • 3 encoder (cross encoder is added cross-attention module, behavior like decoder)
  • 2 linear layer to project output CLS embedding vector into common 256-d vector space
  • 2 MLP for prediction: MLM & ITM

Attibute Prompt Learning (AP) objective¶

Image - Attribute (Prompt) Constrastive Learning¶

Idea from constrastive learning loss, that used in almost papers, e.g. CLIP

• Given a fix set of binary attribute $\mathcal{A_{binary}}$ (27 attribute)
  • Turn this set into fixed set of attribute-prompt: $\mathcal{A_{prompt}}$
  • $|\mathcal{A_{prompt}}| = 2 \cdot |\mathcal{A_{binary}}|$
• Sample pairs of matched ground-true image - attribute prompt $(I, A)$ from a minibatch $\mathcal{B}$ ($\mathcal{B}$ contain positive (text caption -image) pairs):
  • $I$: an image $\in$ $\mathcal{B}$
  • $A$: an attribute prompt $\in$ $\text{fixed-set}$ $\mathcal{A_{prompt}}$
• Call this set of all matched ground-true pair (image - attribute prompt) pairs in a mini-batch as $\mathcal{B_1}$
  • $\mathcal{B_1} = \{(I, A) | I \in \mathcal{B}, A \in \mathcal{A_{prompt}}, I \text{ match } A\}$
• Calculate loss for a matched pair of (image, attribute prompt) in $\mathcal{B_1}$:
  • Based on common contrastive loss $\mathcal{L} = -\log(score), score \in [0,1] $
    
  • How to calculate similarity $score$:
    • Cosine similarity $s$ between positive pair
    • Scale by softmax for all positive and negative pairs
    • Smooth by temperature $\tau$ (learnable parameter in this paper)
      
      $$ \mathcal{S_{i2a}}(I, A) = \frac{ \exp(s(F_I, F_{A^{+}})/ \tau) } {\exp(s(F_I, F_{A^{+}})/ \tau) + \exp(s(F_I, F_{A^{-}})/ \tau)} $$
      
      
  • Score and Loss for Image-Attribute Contrastive Learning:
    • Calculate in only one direction (not bi-direction): image-to-attribute $$ \mathcal{S_{IAC}}(I, A) = \mathcal{S_{i2a}}(I, A)$$
      $$ \mathcal{L_{IAC}}(I, A) = -\log \mathcal{S_{IAC}}(I, A)$$
      
      

• Loss for a minibatch $\mathcal{B}$ which sample $\mathcal{B_1}$ pairs (abbreviate $\mathbb{E}$ for expected value) $$ \mathcal{L_{IAC}}(\mathcal{B}) = \mathbb{E}_{(I, A)\sim \mathcal{B_1}}[-\log \mathcal{S_{IAC}}(I, A)]$$

Image-Attribute Matching Learning (IAM)¶

• From a minibatch $\mathcal{B}$, sample $\mathcal{B_2}$ set of pairs of $(I, A)$ that:

  • $I$: image from minibatch
  • $A$: sample 5 random attribute prompt from $\mathcal{A_{prompt}}$
  • $|\mathcal{B_2}| = 5 \cdot |\mathcal{B}|$

• Calculate loss for each pair $(I, A)$ in $\mathcal{B_2}$:

  • Consider this as binary classfication problem, classsify a pair of image-attribute prompt as matched/not-matched
  • Use a cross-encoder to fuse image embedding and text embedding from two encoders: $$\mathbf{V} = \{v^{cls}, v^1, \dotsc, v^{k_I}\} = \text{VisionEncoder}(I)$$
  
  

  $$\mathbf{L} = \{l^{cls}, l^1, \dotsc, l^{k_L}\} = \text{TextEncoder}(A)$$

  
  

  $$\mathbf{C} = \{c^{cls}, c^1, \dotsc, c^{c_L}\} = \text{CrossEncoder}(\mathbf{V}, \mathbf{L})$$

  
  
  • Final output embedding of [CLS] token, will feed in to a MLP with Sigmoid function, then get matched probability
  
  

  $$p_{matched}(I, A) = Sigmoid(MLP(c^{cls}))$$

  
  
  • Loss: binary cross-entropy
    • $\text{matched}(I, A): \text{ground-true boolean function to check matched or unmatched}$
    • Loss for a pair $(I, A)$: $$\mathcal{L_{IAM}}(I, A) = -\log{p_{matched}(I, A)} \cdot \text{matched}(I, A)$$
  
  

• Final IAM loss for a minibatch $\mathcal{B}$: $$ \mathcal{L_{IAC}}(\mathcal{B}) = \mathbb{E}_{(I, A)\sim \mathcal{B_2}}[\mathcal{L_{IAM}}(I, A)]$$

Masked Attribute Language Modeling (MAL)¶

• Given a fix set of binary attribute prompt $\mathcal{A_{prompt}}$, do this strategy to generate masked prompt:

  1. Mask out the text tokens with a probability of 25%
  2. Among the masked tokens,
     • 10% and 80% is replaced with random tokens and the special token [MASK], respectively
     • 10% remain unchanged.

  $\rightarrow$ $A_{mask} = \text{MASK}(A)$

• Recall $\mathcal{B_1} = \{(I, A) | I \in \text{mini-batch } \mathcal{B}; A \in \mathcal{A_{prompt}}; I \text{ match } A\}$

• For each pair $(I, A)$ in $\mathcal{B_1}$ $\rightarrow$ get $(I, A_{mask})$

  • Generate embedding $\mathbf{V}, \mathbf{\tilde{L}_{A-mask}}$ through vision and text encoder
  • Generate cross embedding $\{c^{cls}_A, c^1_A, \dotsc, c^{N_T}_A\}$ through cross-encoder
  • Consider at classification problem for each mask token:
    • Number of class is the size of vocabulary, include special token
    • Feed each token to a MLP with Softmax activation
    • Use cross-entropy loss to calculate for each masked token

• Final loss for MAL $$ \mathcal{L_{MAL}}(\mathcal{B}) = \mathbb{E}_{token_{mask} \sim (I, A)\text{ / } (I, A) \sim \mathcal{B}}[CELoss(token_{mask})]$$

Final loss for Attribute Prompt Learning¶

$$\mathcal{L_{APL}} = \mathcal{L_{MAL}}(\mathcal{B}) + \mathcal{L_{IAC}}(\mathcal{B}) + \mathcal{L_{IAM}}(\mathcal{B})$$

Text Matching Learning (TM) objective¶

Image-Text Contrastive Learning (ITC)¶

• Given a minibatch $\mathcal{B}$ of images, each image has a ground-true corresponding text captioning
  • For each image, generate a positive pair for each image and $|\mathcal{B}| - 1$ negative pair (image-text captioning)
  • Calculate contrastive score in two way: text-2-image and image-2-text, for each image in minibatch $$ \mathcal{S_{i2t}}(I, T) = \frac{ \exp(s(F_I, F_{T^{+}})/ \tau) } {\exp(s(F_I, F_{T^{+}})/ \tau) + \exp(s(F_I, F_{T^{-}})/ \tau)} $$
    
    $$ \mathcal{S_{t2i}}(T, I) = \frac{ \exp(s(F_T, F_{I^{+}})/ \tau) } {\exp(s(F_T, F_{I^{+}})/ \tau) + \exp(s(F_T, F_{I^{-}})/ \tau)} $$
    
• Final ITC loss for a minibatch: $$ \mathcal{L_{ITC}}(\mathcal{B}) = \mathbb{E}_{(I, T) \sim \mathcal{B}}[\frac {\mathcal{S_{i2t}}(I, T) + \mathcal{S_{t2i}}(T, I))} {2}]$$

Image-Text Matching Learning (ITM)¶

• Loss is calculate like APL (use BCE loss), difference is the stratregy to sample pair from minibatch for binary classify
• Given a minibatch $\mathcal{B}$:
  • Generate $|\mathcal{B}|$ positive pair for each image and corresponing text captioning
  • Generate $2 * |\mathcal{B}|$ hard negative pair, follow this stratregy:
    • For each image, use image-text constrastive to choose 2 hard negative text-captioning by similarity score
      => Consider as classfication problem for classify $|\mathcal{B}|$ matched pair and $2 \cdot |\mathcal{B}|$ unmatched pair

Masked Language Modeling (MLM)¶

• Loss is calculate like APL (use CE loss), difference is that the stratregy to sample pair from minibatch for binary classify:
  • Each image has several matched attribute prompt
  • Each image has a fixed number text captioning

Final pretraining loss¶

• \mathcal{B}: minibatch of positive (text caption - image) pair, 5 attribute prompt is random sampled for each image
• Default hyperparameter $\beta = 0.8$ $$\mathcal{L}(\mathcal{B}) = [\mathcal{L_{ITC}}(\mathcal{B}) + \mathcal{L_{MLM}}(\mathcal{B}) + \mathcal{L_{ITM}}(\mathcal{B})] + \beta \cdot \mathcal{L_{APL}}(\mathcal{B})$$

Final loss for text TBPR¶

• $\mathcal{B}$: minibatch of positive (text caption - text augmentation (EDA) - image) triplet

• Default hyperparameter $\gamma = 0.8$ $$ \mathcal{L}(\mathcal{B}) = [\mathcal{L_{ITC}}(\mathcal{B}) + \mathcal{L_{MLM}}(\mathcal{B}) + \mathcal{L_{ITM}}(\mathcal{B})] + \gamma [\mathcal{L_{ITC-EDA}}(\mathcal{B}) + \mathcal{L_{MLM-EDA}}(\mathcal{B}) + \mathcal{L_{ITM-EDA}}(\mathcal{B})]$$

Input for loss function and How to sampling input?

Training phase¶

Dataset & Data Augmentation¶

For text¶

• EDA: from paper "Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks", EMNLP 2019. For a given sentence in the training set, randomly choose and perform one of the 4 following simple but powerful operations:

1. Synonym Replacement (SR): Randomly choose n words from the sentence that are not stop words. Replace each of these words with one of its synonyms chosen at random.
2. Random Insertion (RI): Find a random synonym of a random word in the sentence that is not a stop word. Insert that synonym into a random position in the sentence. Do this n times.
3. Random Swap (RS): Randomly choose two words in the sentence and swap their positions. Do this n times.
4. Random Deletion (RD): Randomly remove each word in the sentence with probability p

For image¶

• Random horizontal flipping
• RandAugment
  • From paper "Randaugment: Practical automated data augmentation with a reduced search space", CVPR 2020
• Random erasing
  • From paper "Random erasing data augmentation", AAAI 2020

Implemention detail¶

Pretraining

• Pre-train APTM with Pytorch on 4 NVIDIA A100 GPUs for 32 epochs, and the mini-batch size is 150.
• Adopt the AdamW optimizer with a weight decay of 0.01.
• Every image input is resized to 384 × 128
• There are 214.5𝑀 trainable parameters in APTM

Evaluate APTM on finetunning TBPR datasets:

• Dataset evaluate: CUHK-PEDES, RSTPReid, and ICFG-PEDES datasets
• Adopt EDA for text data augmentation and set the mini-batch size as 120.
• In reference, for each text query,
  • First compute its cosine similarity with all images and take the top-128 image candidates.
  • Calculate the matching probability (from ITM head) between the text query and every selected image candidate for ranking.

Evaluate APTM on PAR datasets:

• Compute the matching probability between every image and every pair of attribute prompts for ranking.
• An attribute prompt with a higher matching probability means the image is more relevant to the corresponding attribute.

Only use 0.3/1.5M images to get SOTA benchmark

Conclusion¶

Demo in notebook¶

Set up¶

Define path¶

Import libries / local modules¶

Load config for MALS pretraining¶

Load model checkpoint¶

Bert Tokenizer & Mask Generator¶

Build dataset loader¶

Inference¶

Preprocess batch input before feeding into train model

Feed data into train model

Print model architecture¶

Loss¶

Data Augmentation¶