Upload paper/paper.ltx with huggingface_hub

paper/paper.ltx

@@ -0,0 +1,803 @@
+\documentclass[12pt,a4paper]{article}
+\usepackage[utf8]{inputenc}
+\usepackage{amsmath, amssymb}
+\usepackage{graphicx}
+\graphicspath{{../figures/}}
+\usepackage{booktabs}
+\usepackage{caption}
+\usepackage{float}
+\usepackage{subcaption}
+\usepackage{hyperref}
+\usepackage{geometry}
+% \usepackage{multirow}  % not used
+
+\geometry{margin=1in}
+
+\title{GAP (Guaranteed Attribute Position) CLIP: A Multi Loss Framework for Attribute Aware Fashion Embeddings}
+\author{}
+\date{}
+
+\begin{document}
+
+\maketitle
+
+\begin{abstract}
+Embeddings are powerful but partial representations. They capture broad semantic relationships, but they’re not exhaustive or perfectly faithful to human meaning. We propose \textbf{GAP CLIP (Guaranteed Attribute Position CLIP)}, a modular extension of the CLIP framework designed for \textbf{fashion retrieval and attribute aware embeddings}. Unlike standard CLIP, which produces generic representations, GAP CLIP explicitly allocates dimensions to key fashion attributes — a 16 dimensional color module and a 64 dimensional category module — integrated into the CLIP backbone. This design yields embeddings that are both interpretable and semantically structured.
+
+Across internal and external datasets (KAGL Marqo \cite{marqo}, Fashion MNIST \cite{mnist}), GAP-CLIP achieves several-fold improvements in embedding separation relative to the Fashion CLIP baseline \cite{baseline}, with relative gains ranging from roughly +240\% to more than +1{,}000\% depending on modality and dataset. This confirms a markedly superior disentanglement of attribute semantics. Despite its modularity, GAP CLIP maintains competitive retrieval precision and strong generalization, while offering compact and interpretable embedding subspaces. These results highlight that \textbf{embedding separability}—rather than raw accuracy—can be a decisive advantage for semantic search and explainable multimodal understanding in specialized domains such as fashion.
+
+
+By design, GAP CLIP trades some raw classification accuracy for dramatically improved embedding structure: the specialized sub-models use minimal encoders (e.g., 38-token vocabulary, 16-d projection for color) that are orders of magnitude smaller than the baseline’s full CLIP encoder. Where accuracy does decrease, the loss is concentrated in text-based classification—a task where the baseline benefits from its large-capacity text encoder rather than from superior visual understanding. Image-based accuracy and, crucially, separation scores consistently favor GAP CLIP, confirming that the approach offers a compelling balance between interpretability and retrieval-oriented performance.
+
+\end{abstract}
+
+\section{Introduction}
+Recent advances in multi modal learning have demonstrated the power of jointly modeling vision and language. Among them, CLIP~\cite{clip}, trained on large scale image–text pairs, has emerged as a state of the art framework for vision–language alignment. Its ability to generalize across unseen tasks enables zero shot classification and open vocabulary retrieval, making it a versatile foundation for many applications.
+
+While prior works emphasize retrieval precision or classification accuracy, our findings show that \textbf{embedding separability}—the structural organization of semantic spaces—can be a more consistent and informative signal of model quality. GAP CLIP is therefore positioned not as a model that merely retrieves better, but as one that \textbf{organizes semantic attributes more effectively}, enabling superior downstream retrieval and interpretability.
+
+
+However, general purpose models like CLIP often fail to capture fine grained attributes that are crucial in specialized domains. In fashion, attributes such as color, category, and style play a central role in retrieval and recommendation~\cite{liu2016deepfashion, hsiao2018learning}. Distinguishing between navy blue and royal blue, or between leggings and skinny pants, requires more than generic semantic alignment—it calls for dedicated attribute modeling.
+
+To address this limitation, we introduce GAP CLIP, a modular extension of CLIP that integrates specialized embedding models for fashion attributes. Our contributions are threefold:
+\begin{itemize}
+    \item We design a modular embedding architecture that dedicates 16 dimensions to color and 64 dimensions to category, while preserving 432 dimensions for standard CLIP features.
+    \item We demonstrate that this modular design significantly enhances retrieval-oriented structure, yielding several-fold improvements in separation scores over baseline CLIP \cite{baseline} (often between 3$\times$ and 11$\times$ higher separation) across datasets and modalities.
+    \item We provide structural analyses showing high attribute separation and robust generalization across both internal datasets and external benchmarks.
+\end{itemize}
+
+By explicitly structuring the embedding space, GAP CLIP delivers both interpretability and performance, paving the way for attribute aware multi modal systems in fashion. GAP CLIP is therefore positioned not as a model that merely retrieves better, but as one that structures semantic attributes more effectively.
+
+\section{Related Work}
+
+\subsection{Vision–Language Pre-training}
+Contrastive vision–language frameworks such as CLIP~\cite{clip} and ALIGN~\cite{jia2021scaling} have achieved remarkable success in aligning textual and visual representations. These models leverage large scale weakly paired image–text data to learn embeddings applicable to a wide range of downstream tasks~\cite{phang2022multimodal}. However, they are designed to be domain agnostic and lack mechanisms for modeling specific attributes such as color or category.
+
+\subsection{Fashion Understanding}
+Early works in fashion focused on attribute prediction and category classification. The DeepFashion dataset~\cite{liu2016deepfashion} introduced benchmarks for clothing category and attribute recognition, fueling research on convolutional architectures for fine grained analysis~\cite{ak2022attribute}. More recent works explore multi task learning for predicting attributes such as category, texture, and style~\cite{chen2020fashion, li2020learning}. Despite their effectiveness, these models generally rely on monolithic representations, limiting interpretability and modularity.
+
+\subsection{Modular Architectures}
+The concept of modular specialization has been explored in other domains. Li et al.~\cite{li2022modular} proposed modular encoders tailored to anatomy and pathology in medical imaging, improving interpretability. Similarly, Zhou et al.~\cite{zhou2023geospatial} applied modular vision–language alignment to geospatial applications. To the best of our knowledge, such modular extensions have not been systematically applied in fashion. Our work addresses this gap by designing a hybrid modular CLIP architecture that combines domain specific embeddings with general purpose representations.
+
+\section{Methodology}
+
+\begin{figure}[h!]
+\centering
+    \includegraphics[width=\textwidth]{scheme.png}
+    \caption{Main methodology}
+    \label{schema}
+\end{figure}
+
+
+\subsection{Overall Architecture}
+Our architecture enhances the CLIP framework by incorporating two specialized attribute modules: color and category. These modules are integrated into a unified multimodal representation space. The final 512 dimensional embedding comprises 16 dimensions for color, 64 for category, and 432 from the original CLIP backbone. This design strikes a balance between compact attribute encoding and general purpose semantic features, offering both interpretability and strong performance (Figure~\ref{schema}).
+
+This modular approach involves independently pre-training specialized feature models before concatenating them into the main CLIP model. This allows for efficient training and enables the model to utilize domain specific knowledge while preserving the generalizability of the CLIP backbone. It also facilitates convergence and reduces overfitting.
+
+\subsection{Color Model}
+The color model introduces a specialized multimodal architecture tailored for fine grained color recognition in fashion imagery, extending the CLIP framework with domain specific constraints and optimizations. It employs a dual encoder design—one for images and one for text—projecting both modalities into a compact 16 dimensional attribute specific embedding space.
+
+Architecture Details:
+\begin{itemize}
+\item Image Encoder: ResNet-18 (pretrained) with dropout (0.1) and projection head $512 \rightarrow 16$ + L2 normalization
+\item Text Encoder: Custom embedding layer (vocab size = 38), dropout (0.1), length weighted pooling, projection $32 \rightarrow 16$ + L2 normalization
+\item Tokenization: Custom color aware tokenizer extracting only color relevant keywords
+\item Total Parameters: $\sim$45M
+\item Embedding Dimension: 16
+\end{itemize}
+
+Unlike traditional CLIP implementations relying on BERT style tokenization, the color model uses a compact, domain specific vocabulary of 38 tokens focused exclusively on color semantics. This reduces linguistic noise and improves alignment between visual and textual representations for fashion attributes.
+
+The model is trained using a symmetric contrastive objective:
+
+\[ 
+L = (L_{i2t} + L_{t2i})/2
+\]
+
+where $L_{i2t}$ and $L_{t2i}$ denote image-to-text and text-to-image contrastive cross entropy losses respectively. A temperature parameter $\tau = 0.07$ controls similarity scaling.
+
+Training is performed over 50 epochs using Adam with a learning rate of $1\mathrm{e}{-4}$, weight decay $1\mathrm{e}{-5}$, and a batch size of 16. The dataset consists of 11,000 curated fashion images paired with text descriptions and color labels across 11 categories (beige, black, blue, brown, green, orange, pink, purple, red, white, yellow), using a 90/10 train test split.
+
+Regularization is ensured through encoder dropout (0.1) and optimizer weight decay, preventing overfitting and reinforcing generalization. The resulting 16 dimensional embedding space enables highly efficient similarity computation and real time inference while retaining strong discriminative power for color classification tasks.
+
+\subsection{Category Model}
+The category model employs a hierarchical classification approach, leveraging a ResNet-18 for image encoding and a lightweight text encoder. This allows for robust category identification from natural language descriptions, achieved through rule based regular expression mappings. 
+
+\noindent\textbf{Architecture Details:}
+\begin{itemize}
+    \item Image Encoder: ResNet-18 with custom projection head $512 \rightarrow 128 \rightarrow 64$
+    \item Text Encoder: Embedding layer followed by projection (vocab\_size $\rightarrow$ 64)
+    \item Total Parameters: $\sim$45M
+    \item Embedding Dimension: 64
+\end{itemize}
+
+The category model is trained using a multi objective loss function, which combines classification, consistency, and contrastive objectives:
+\[
+L_\text{category} = 1.0 \, L_\text{classification} + 0.3 \, L_\text{consistency} + 0.2 \, L_\text{contrastive}
+\]
+
+This formulation ensures accurate category classification (given primary importance with a weight of 1.0), consistency between image and text representations (weighted at 0.3), and contrastive alignment for enhanced embedding quality (weighted at 0.2).
+
+\subsection{Main CLIP Integration}
+
+Conventional multimodal models frequently yield black box embeddings that lack interpretability. To address this limitation, we introduce an architecture that explicitly allocates dimensions to semantic attributes, thus enabling both high performance and interpretability.
+The proposed model builds on the CLIP framework~\cite{clip} and incorporates pretrained modules for color and category attributes. The base architecture is the open-source LAION CLIP ViT-B/32~\cite{laion}, producing 512-dimensional embeddings. The pretrained attributes models for color and category are concatenated with the CLIP features to yield the final 512 dimensional representation.
+
+The training objective combines two complementary components. The first is a triple contrastive loss aligning text, image, and attribute representations:
+
+\[
+L_\text{contrastive} = 0.7 \, L_\text{text-image} + 0.15 \, L_\text{text-attr} + 0.15 \, L_\text{image-attr}
+\]
+
+The second is a direct attribute alignment loss that enforces the first 80 dimensions of the main model to match the outputs of the pretrained specialized modules. Concretely, for both text and image modalities, we compute MSE and cosine similarity losses between the color subspace (dims 0--15) and the color model embeddings, and between the category subspace (dims 16--79) and the category model embeddings:
+
+\[
+L_\text{align} = \frac{1}{2}\bigl(L_\text{color-align} + L_\text{category-align}\bigr)
+\]
+
+where each alignment term combines an MSE loss and a cosine dissimilarity. To prevent catastrophic forgetting, we additionally include a reference loss $L_\text{ref}$ that penalizes deviation from the original pretrained CLIP text embeddings (MSE between the fine-tuned and frozen reference text features). The final loss is:
+
+\[
+L_\text{total} = (1 - \alpha) \, L_\text{contrastive} + \alpha \, L_\text{align} + \beta \, L_\text{ref}
+\]
+
+with alignment weight $\alpha$ and reference weight $\beta$ tuned via hyperparameter search (we use $\beta = 0.1$). This design is the key mechanism behind Guaranteed Attribute Positioning (GAP): it directly supervises specific dimensions using expert sub-models rather than relying solely on contrastive signals.
+
+We automated hyperparameter search for the main CLIP model using Optuna~\cite{optuna}. Each trial sampled learning rate, temperature, alignment weight, and weight decay, trained a fresh CLIP model for 5 epochs on a 5,000-sample subset, and evaluated the validation loss. Optuna pruned weak trials early and selected the best trial in terms of validation loss, with learning rate $1.42 \times 10^{-5}$, temperature $0.0503$, alignment weight $0.5639$, and weight decay $2.76 \times 10^{-5}$. We then retrained the model end-to-end with these best hyperparameters on a 20,000-sample subset, saving the resulting checkpoint and training curves for downstream evaluation. The hyperparameter importances are shown in Figure~\ref{optuna}. The learning curves are in Figure~\ref{training_curves}.
+
+
+\section{Datasets and Experimental Setup}
+\begin{itemize}
+    \item \textbf{Internal dataset}: 436,985 fashion items (10,000 used for evaluation, sampled via URL-based image retrieval) with 35 colors, 20+ categories, 80/20 train/validation split; in-domain evaluation on the same distribution as training.
+    \item \textbf{KAGL Marqo}~\cite{marqo}: 44,434 items, 46 colors, 44 categories; external test only, real-world fashion e-commerce data with diverse categories.
+    \item \textbf{Fashion MNIST}~\cite{mnist}: 10,000 grayscale test images, 10 categories; external test only, standard benchmark with clear category structure.
+\end{itemize}
+
+Our internal fashion dataset comprises 436,985 items, each with an image, textual description, color annotation, and category label. For the main GAP-CLIP model we trained on a 20,000-sample subset (80/20 train/validation split, stratified sampling) drawn from this pool; the standalone color model was trained on 11,000 curated images. Each sample includes: high-resolution product images (224$\times$224 pixels after preprocessing), rich textual descriptions containing product details, materials, and attributes, color labels covering 11 distinct color categories, and category labels spanning 20+ fashion categories (accessories, bag, dress, shoes, top, etc.).
+Images underwent standard augmentation during training including random horizontal flips (50\% probability), rotation ($\pm$15\textdegree{}), color jittering (brightness, contrast, saturation, hue variations), Gaussian blur, and random affine transformations. Validation and test images used only resizing and normalization.
+We compared our model against the Fashion CLIP model~\cite{baseline}, which was trained on fashion-specific data. While the exact training dataset size for Fashion CLIP is not publicly disclosed, it represents a strong baseline in the fashion domain.
+
+\section{Results}
+
+The metrics used are: (1) the nearest-neighbor classification accuracy in embedding space, which assigns each sample the label of its most similar neighbor by cosine similarity; and (2) the separation score, defined as the difference between the mean intra-class cosine similarity and the mean inter-class cosine similarity (higher values indicate better class separation, meaning items of the same class cluster more tightly than items of different classes).
+
+\subsection{Color Model}
+
+
+\begin{table}[H]
+\centering
+\caption{Performance comparison between CLIP baseline and our 16-d color model. The internal dataset uses 10{,}000 items (11 color classes). Gain is the relative improvement $(\text{GAP-CLIP} - \text{Baseline})/|\text{Baseline}|$.}
+\vspace{0.5em}
+\begin{tabular}{@{}lccc@{}}
+\toprule
+\multicolumn{4}{c}{\textbf{Internal Dataset}} \\
+\midrule
+\textbf{Metric} & \textbf{Baseline} & \textbf{GAP-CLIP} & \textbf{Gain (\%)} \\
+\midrule
+Text embeddings accuracy          & 97.0\%  & 85.6\%  & $-$11.75\%   \\
+Image embeddings accuracy         & 64.5\%  & 70.3\%  & +8.99\%  \\
+Separation score text embeddings  & 0.1562  & 0.5136  & +228.81\% \\
+Separation score image embeddings & 0.0356  & 0.2994  & +741.01\% \\
+\midrule
+\multicolumn{4}{c}{\textbf{KAGL Marqo Dataset}} \\
+\midrule
+Text embeddings accuracy          & 92.4\%  & 87.5\%  & -5.30\%   \\
+Image embeddings accuracy         & 57.1\%  & 64.5\%  & +13.00\%  \\
+Separation score text embeddings  & 0.1500  & 0.5895  & +293.00\% \\
+Separation score image embeddings & 0.0395  & 0.2743  & +594.43\% \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+The comparative evaluation between the CLIP baseline and our color model on the internal dataset (10{,}000 items, 11 color classes) reveals an important asymmetry. The baseline achieves higher text accuracy (97.0\% vs.\ 85.6\%), but this comparison is inherently unequal: the baseline encodes text with a full CLIP text encoder (77M parameters, 512 dimensions), for which recognizing a color is essentially a reading-comprehension task—the color name appears literally in the product description. Our color model, by contrast, uses a minimal text encoder (38-token vocabulary, 32-d embedding projected to 16-d, roughly 2k parameters) that extracts only color-relevant keywords. Given this 4-orders-of-magnitude difference in model capacity, 85.6\% text accuracy is a strong result for the specialized encoder.
+
+Crucially, the metrics that matter most for structured retrieval tell a different story. On \textbf{image accuracy}—where the model must actually perceive color from pixels—our model improves from 64.5\% to 70.3\% (+8.99\%). The separation scores show the decisive advantage: text separation increases from 0.1562 to 0.5136 (+228.81\%) and image separation from 0.0356 to 0.2994 (+741.01\%), demonstrating far superior attribute disentanglement in a space that is 32$\times$ more compact. For a fashion search engine, high separation means that queries like ``find me this item in blue'' produce tightly clustered, relevant results—a property that raw nearest-neighbor accuracy does not capture.
+
+On the KAGL Marqo dataset (7{,}643 items after color filtering), the picture is consistent: image accuracy improves from 57.1\% to 64.5\% (+13.00\%), and both separation scores increase substantially—text separation from 0.1500 to 0.5895 (+293.00\%) and image separation from 0.0395 to 0.2743 (+594.43\%). Text classification accuracy decreases from 92.4\% to 87.5\% ($-$5.30\%), again reflecting the capacity asymmetry rather than a modeling weakness.
+
+Overall, these results highlight that our color model produces highly structured and separable color embeddings while improving image accuracy over the full-dimensional baseline. The text accuracy gap is an expected consequence of the deliberate architectural choice to use a minimal, interpretable text encoder; it does not diminish the model's practical value for color-aware retrieval. Note that Fashion-MNIST is excluded from the color evaluation as it contains only grayscale images without color annotations. The confusion matrices in Figure~\ref{fig:color_cm_local} provide per-class classification breakdowns for the color model on the internal test dataset.
+
+
+\subsection{Category Model}
+
+\begin{table}[H]
+\centering
+\caption{Comprehensive performance comparison between CLIP baseline and our category model, including percentage gains recalculated using relative improvement. Separation scores and nearest neighbor accuracy are reported for test, Fashion MNIST, and KAGL Marqo datasets.}
+\label{tab:model_comparison}
+\vspace{0.5em}
+\begin{tabular}{@{}lccc@{}}
+\toprule
+\multicolumn{4}{c}{\textbf{Test Dataset}} \\
+\midrule
+\textbf{Metric} & \textbf{Baseline} & \textbf{GAP-CLIP} & \textbf{Gain (\%)} \\
+\midrule
+Text embeddings accuracy          & 74.8\%  & 100.0\% & +33.69\%  \\
+Image embeddings accuracy         & 65.6\%  & 77.2\%  & +17.68\%  \\
+Separation score text embeddings  & 0.0834  & 0.9103  & +991.37\% \\
+Separation score image embeddings & 0.0582  & 0.6496  & +1016.15\% \\
+\midrule
+\multicolumn{4}{c}{\textbf{KAGL Marqo Dataset}} \\
+\midrule
+Text embeddings accuracy          & 95.9\%  & 98.8\%  & +2.98\%   \\
+Image embeddings accuracy         & 95.1\%  & 92.8\%  & -2.30\%   \\
+Separation score text embeddings  & 0.1551  & 0.7667  & +394.33\% \\
+Separation score image embeddings & 0.1318  & 0.5233  & +297.04\% \\
+\midrule
+\multicolumn{4}{c}{\textbf{Fashion MNIST Dataset}} \\
+\midrule
+Text embeddings accuracy          & 100\%   & 100\%   & 0.00\%    \\
+Image embeddings accuracy         & 90.2\%    & 85.7\%  & -4.52\%  \\
+Separation score text embeddings  & 0.0826  & 0.9531  & +1053.87\% \\
+Separation score image embeddings & 0.0572  & 0.1965  & +243.53\% \\
+\bottomrule
+\end{tabular}
+\vspace{0.5em}
+\end{table}
+
+
+The extended evaluation across three datasets shows that our category model delivers consistent and substantial gains in embedding separability, with relative separation improvements ranging from +243.53\% to +1{,}053.87\% depending on modality and dataset. On the test dataset, the model achieves perfect text classification (100.0\% vs.\ 74.8\%, +33.69\%) and higher image accuracy (77.2\% vs.\ 65.6\%, +17.68\%), while separation scores increase sharply for both text (+991.37\%) and images (+1{,}016.15\%).
+
+On the KAGL Marqo dataset, text and image separation also improve strongly (+394.33\% and +297.04\%, respectively), indicating that the structured embedding space generalizes to a real-world ecommerce benchmark. In this setting, text accuracy improves from 95.9\% to 98.8\% (+2.98\%), while image accuracy decreases from 95.1\% to 92.8\% (-2.30\%).
+
+On Fashion-MNIST, text accuracy remains saturated at 100.0\% for both models, image accuracy decreases from 90.2\% to 85.7\% (-4.52\%), and separation still improves markedly for text (+1{,}053.87\%) and images (+243.53\%).
+
+Overall, these results indicate that the proposed architecture consistently produces a more structured, attribute-aware category embedding space than the CLIP baseline, while maintaining competitive accuracy across datasets.
+
+The confusion matrices in Figures~\ref{fig:cat_cm_local},~\ref{fig:cat_cm_kaggle}, and~\ref{fig:cat_cm_fashion} provide per-class classification breakdowns for the category model on the internal, KAGL Marqo, and Fashion-MNIST datasets, comparing GAP-CLIP against the baseline.
+
+\subsection{Main CLIP Model}
+
+\subsubsection{Evaluation}
+
+We conducted several evaluation tests. The first is the classification accuracy, the second the separation score, then the heatmaps to compare the similarity between the embedding parts (Figure~\ref{heatmap}), the t-SNE~\cite{tsne} approach (Figure~\ref{tsne}), the simulation of a search engine on our database to check that the embeddings were meaningful (Figure~\ref{search_engine}), and finally the semantic evaluation.
+
+These contributions were validated through a rigorous multi-dataset evaluation on Fashion-MNIST, the Kaggle Fashion dataset, and real-world data, demonstrating the robustness, interpretability, and transferability of the proposed approach.
+
+The results reported in Table~\ref{tab:results} thus provide a unified view of how GAP CLIP performs across tasks, datasets, and embedding granularity.
+
+\subsubsection{Results}
+
+\subsubsection{Classification accuracy}
+
+To assess the quality of the learned embedding space, we evaluate classification performance using a nearest-neighbor accuracy metric, which measures the ability of embeddings to cluster samples from the same class. For each sample, we compute cosine similarities to all other embeddings, exclude self-similarity, and assign the label of the most similar neighbor. The final accuracy corresponds to the proportion of samples whose nearest neighbor shares the same ground-truth class. This metric provides a model-free evaluation of the embedding geometry: high accuracy indicates that samples of the same class form compact, well-separated clusters, while lower accuracy reveals overlap or insufficient disentanglement between classes. Because it relies solely on distances in the embedding space, this approach offers an unsupervised yet sensitive probe of local structure, without requiring a trained classifier. However, it can be affected by outliers or class imbalance, which may distort nearest-neighbor relationships. Overall, nearest-neighbor accuracy provides an interpretable and robust measure of how effectively the embedding space captures semantic similarity. \\
+
+To resolve the inherent tension between using specialized embeddings (interpretable but potentially less performant) and full embeddings (high-performing but opaque), we propose a hybrid ensemble system that strategically combines specialized and full embeddings. For external datasets (KAGL Marqo), we use 40\% specialized embeddings with 60\% full embeddings; for the internal test set and Fashion MNIST, we rely on the specialized subspace directly. This mixture preserves the interpretability afforded by the semantic subspaces while retaining the strong predictive performance of the full embedding representation. In our evaluations, this ensemble strategy proved particularly effective, maintaining competitive accuracy while enabling meaningful dimensional inspection.
+
+\begin{table}[H]
+\centering
+\small
+\caption{Comparison of the accuracies of the baseline embeddings and GAP-CLIP model embeddings, including relative gains in percentage.}
+\vspace{0.5em}
+\begin{tabular}{@{}lccc@{}}
+\toprule
+\textbf{Test Dataset} \\
+\midrule
+\textbf{Metric} & \textbf{Baseline} & \textbf{GAP-CLIP} & \textbf{Gain (\%)} \\
+\midrule
+Color acc (text)   & 83.4\% & 97.3\% & +16.67\% \\
+Color acc (image)  & 42.3\% & 64.7\% & +52.96\% \\
+Cat acc (text)     & 68.1\% & 92.7\% & +36.12\% \\
+Cat acc (image)    & 68\%   & 76.5\% & +12.50\% \\
+\midrule
+\textbf{KAGL Marqo Dataset} \\
+\midrule
+Color acc (text)   & 83.6\% & 84\%  & +0.48\% \\
+Color acc (image)  & 26\%   & 43\%  & +65.38\% \\
+Cat acc (text)     & 97.4\% & 98.4\% & +1.03\% \\
+Cat acc (image)    & 89.2\% & 90\%  & +0.90\% \\
+\midrule
+\textbf{Fashion MNIST Dataset} \\
+\midrule
+Cat acc (text)     & 100\% & 100\% & 0.00\% \\
+Cat acc (image)    & 85\%  & 82\%  & -3.53\% \\
+\bottomrule
+\end{tabular}
+\label{tab:results}
+\end{table}
+
+
+Across all evaluated datasets, our subspace-based model consistently outperforms the baseline in classification accuracy for most settings, particularly where fine-grained semantic disentanglement is crucial. On the primary test dataset, the proposed model achieves notable gains in both color and category prediction, with relative improvements of +16.67\% in text-based color classification, +52.96\% in image-based color prediction, +36.12\% in text-based category prediction, and +12.50\% in image-based category prediction. These systematic increases demonstrate that structuring the embedding space into semantic subspaces enhances both vision and language alignment.
+
+The benefits are also evident on the KAGL Marqo dataset, which introduces greater visual variability. Here, our model yields a substantial relative improvement of +65.38\% in image-based color classification and smaller yet positive gains in category accuracy (+1.03\% for text and +0.90\% for image), while text-based color accuracy remains essentially saturated. This confirms the robustness of the approach under realistic ecommerce conditions.
+
+On Fashion MNIST, where text-based category accuracy is already saturated at 100\%, our method maintains equivalent performance. Image-based accuracy decreases moderately from 85\% to 82\% (–3.53\% relative change), an effect that appears isolated to this highly stylized grayscale dataset and does not affect performance on more complex real-world distributions. \\
+
+While the baseline provides strong baselines for fashion category recognition by leveraging shape-driven and semantically aligned representations, it does not explicitly model color as a continuous and metric attribute. Our model addresses this limitation by introducing a dedicated color embedding space jointly learned from image and text modalities, optimized with a color-aware loss that enforces perceptual consistency in RGB/Lab space. As a result, the proposed approach preserves the baseline robustness to category-level visual cues while enabling fine-grained and metrically meaningful color representation, leading to improved performance on color-sensitive tasks where standard CLIP-based baselines fail to capture perceptual color distances. \\
+
+Although the training data originate from the same sub-models, the main model benefits from a broader diversity of negative samples arising from the combination of color, hierarchy, and category axes. This variety enriches the structure of the text embedding space and promotes stronger class separation, which can account for its superior performance on general tasks. In contrast, the sub-models are optimized for narrowly targeted objectives and therefore receive a limited training signal tied to a single axis; when evaluated on a more general textual accuracy task involving more classes or varied formulations, this restricted supervision hinders their ability to generalize. Thus, despite relying on identical data, narrow supervision leads sub-models to develop a specialized representation space, whereas the multi-axis supervision of the main model yields a more generalizable text space, better suited for overall accuracy. \\
+
+Overall, these results demonstrate that our subspace-guided representation framework improves interpretability while delivering competitive—and often substantially superior—classification accuracy compared to the baseline, particularly in scenarios requiring fine-grained attribute discrimination. Note that the baseline was trained on billions of images, making it more robust.
+
+
+\subsubsection{Separation score}
+
+Across all evaluations, we measure how well different embedding configurations disentangle color and category attributes in both text and image modalities.
+
+The separation score measures how well an embedding isolates an attribute by comparing the mean intra-class cosine similarity (how tightly same-class items cluster) with the mean inter-class cosine similarity (how well different classes are distinguished). A higher score indicates stronger attribute disentanglement.
+
+In the Fashion-MNIST dataset, in the original 512-dimensional space, we observe near-zero separation for color but moderate separation for category (0.4109 text; 0.1974 image). Crucially, low-dimensional subspaces (16--64 dimensions) reproduce the full-space behavior almost exactly, indicating that the relevant factors of variation are already linearly accessible in compact subspaces and that dimensionality is not a limiting factor for these attributes.
+
+The Kaggle KAGL dataset exhibits substantially improved disentanglement:
+
+\begin{itemize}
+    \item Color: The specialized model achieves very high separation (0.8352 text; 0.3923 image), far outperforming both the baseline 512D model (0.1708; 0.0406) and our own full-dimensional model (0.3407; 0.1290).
+    \item Category: The subspace model provides the strongest image-level separation (0.5613), and its text separation (0.3068) also exceeds both baseline and full-model results ($\sim$0.12--0.18).
+\end{itemize}
+
+Overall, imposing structured subspaces leads to consistently stronger attribute disentanglement, especially for color. The local validation set confirms these trends: the color model achieves separation scores of 0.5136 (text) and 0.2994 (image), vastly outperforming the baseline (0.1562 and 0.0356, respectively). Regarding category, the GAP-CLIP hierarchy subspace achieves 0.9103 (text) and 0.6496 (image), compared to the baseline's 0.0834 and 0.0582—representing roughly tenfold improvements in class disentanglement.
+
+Structured subspaces reliably improve attribute disentanglement, particularly on the more varied Kaggle dataset, where gains over both baseline and full-dimensional models are substantial. Low-dimensional subspaces preserve the disentanglement capacity of the original embedding, suggesting that the relevant semantics are localized and can be isolated during training. Color separation benefits the most from subspace training, while category separation improves primarily on the image side but also shows moderate text-level gains.
+
+\subsubsection{Semantic evaluation}
+
+We defined another test to evaluate the robustness of the GAP-CLIP model against varying text complexity. We conducted zero-shot classification experiments comparing our approach with the baseline. We designed two experimental conditions: (1) simple queries consisting of basic color–object pairs (e.g., "red dress", "blue jeans"), and (2) extended descriptions (see Annex~\ref{long_desc}) containing detailed product attributes including fabric composition, style elements, fit characteristics, and structural details (e.g., "a woman wearing a red dress with a v neckline, made of silk chiffon fabric with a flowing drape, featuring a midi length A-line silhouette, three-quarter sleeves, and delicate lace trim details along the hem"). In addition to garment classification, we also explicitly tested color prediction accuracy under both conditions to assess the sensitivity of each model to color cues in the text. \\
+
+For simple descriptions, both models achieved comparable performance in classifying garment types and colors with near-perfect accuracy. However, when presented with extended descriptions typical of ecommerce product listings, a significant performance gap emerged. The baseline model exhibited substantial degradation in classification accuracy, frequently misclassifying garment categories while maintaining only partial success in color identification despite color information being clearly stated. In contrast, our GAP-CLIP model maintained consistent high accuracy across both simple and complex descriptions, correctly identifying both color and garment type attributes. This robustness to description complexity can be attributed to our model���s specialized embedding structure, which explicitly encodes color (16 dimensions) and hierarchy information (64 dimensions) alongside standard CLIP embeddings (512 dimensions), enabling more stable feature extraction when confronted with verbose, multi-attribute text inputs. To quantify these effects, we found a performance gap of 40\% in favor of GAP-CLIP. These results demonstrate that GAP-CLIP is better suited for real-world fashion search applications where product descriptions are typically detailed and contain multiple overlapping semantic attributes.
+
+
+\subsubsection{Embedding structure evaluation}
+
+
+We conducted four tests, A, B, C, and D to evaluate complementary aspects of the embedding structure and zero-shot behavior of GAP-CLIP, with a specific focus on showing that compact subspaces are not only sufficient but better targeted than the full embedding for attribute-specific comparisons.
+
+In \textbf{Test A} (different colors, same category), each pair must satisfy two conditions: (1)~the 64-d hierarchy cosine similarity must exceed the full 512-d cosine similarity ($\mathrm{sim}_\mathrm{hier} > \mathrm{sim}_\mathrm{full512}$), and (2)~the hierarchy similarity must exceed the color similarity ($\mathrm{sim}_\mathrm{hier} > \mathrm{sim}_\mathrm{color}$). GAP-CLIP passed perfectly on both sub-conditions (1000/1000 each), while the Fashion-CLIP baseline reached 927/1000 overall (92.70\%), with 998/1000 for $\mathrm{sim}_\mathrm{hier} > \mathrm{sim}_\mathrm{full512}$ and 928/1000 for $\mathrm{sim}_\mathrm{hier} > \mathrm{sim}_\mathrm{color}$. GAP-CLIP also showed much stronger relative shifts: $\Delta_\mathrm{hier\,vs\,full}=+111.71\%$ and $\Delta_\mathrm{color\,vs\,full}=-114.67\%$, compared with baseline values of $+22.44\%$ and $-31.17\%$, respectively. Full-embedding pair discrimination remained high for both models (99.80\% GAP-CLIP vs.\ 97.10\% baseline).
+
+In \textbf{Test B} (same color, different categories), each pair must satisfy the inverse conditions: (1)~the 16-d color cosine similarity must exceed the full 512-d similarity ($\mathrm{sim}_\mathrm{color} > \mathrm{sim}_\mathrm{full512}$), and (2)~the color similarity must exceed the hierarchy similarity ($\mathrm{sim}_\mathrm{color} > \mathrm{sim}_\mathrm{hier}$). GAP-CLIP again passed perfectly (1000/1000 on both sub-conditions), whereas the baseline dropped to 30/1000 overall (3.00\%), with 151/1000 for $\mathrm{sim}_\mathrm{color} > \mathrm{sim}_\mathrm{full512}$ and 30/1000 for $\mathrm{sim}_\mathrm{color} > \mathrm{sim}_\mathrm{hier}$. The color subspace shift for GAP-CLIP was +83.39\% vs.\ full embedding (baseline: $-24.09\%$), while the hierarchy shift was $-21.09\%$ for GAP-CLIP (baseline: $+15.88\%$). Full-embedding pair discrimination was similarly high for both models (100.00\% GAP-CLIP vs.\ 99.80\% baseline), showing that the gap comes from attribute localization rather than generic similarity strength.
+
+In \textbf{Test C} (subspace decomposition consistency), we evaluate whether the subspaces of a composite embedding align with isolated attribute encodings. For each of 1\,000 examples, we encode a full description (e.g.\ ``red dress in cotton''), the standalone color (``red''), and the standalone category (``dress''). We then compute four cosine similarities: (1)~\emph{color-match}: $\cos(\mathbf{e}_\mathrm{full}[{:}16],\;\mathbf{e}_\mathrm{color}[{:}16])$, (2)~\emph{color-cross}: $\cos(\mathbf{e}_\mathrm{full}[{:}16],\;\mathbf{e}_\mathrm{hier}[{:}16])$, (3)~\emph{hier-match}: $\cos(\mathbf{e}_\mathrm{full}[16{:}80],\;\mathbf{e}_\mathrm{hier}[16{:}80])$, and (4)~\emph{hier-cross}: $\cos(\mathbf{e}_\mathrm{full}[16{:}80],\;\mathbf{e}_\mathrm{color}[16{:}80])$. A sample passes when both $\text{color-match} > \text{color-cross}$ and $\text{hier-match} > \text{hier-cross}$. GAP-CLIP achieved a 96.20\% overall pass rate, with 100\% on the color sub-condition and 96.20\% on the category sub-condition. Mean similarities further illustrate the separation: color-match averaged 0.9863 versus only 0.1410 for color-cross, and hier-match averaged 0.7966 versus 0.4821 for hier-cross. The baseline model reached only 42.30\% overall (59.60\% color, 82.20\% category), with mean color-match and color-cross nearly indistinguishable (0.4429 vs.\ 0.3824) and similarly for category (0.7756 vs.\ 0.7055). These results confirm that GAP-CLIP embeddings are genuinely compositional: the color subspace of a full description faithfully reproduces the standalone color encoding, and the category subspace faithfully reproduces the standalone category encoding---a property entirely absent from the baseline.
+
+Although full-embedding discrimination remained very strong for both models (A: 99.80\% vs.\ 97.10\%, B: 100.00\% vs.\ 99.80\%), the subspace checks clearly favor GAP-CLIP. These results directly support the key claim: 16 dimensions are enough and better for color, and 64 dimensions are enough and better for category, than relying on a distributed 512-d representation for these targeted signals. \\
+
+In Test D, we assess zero-shot vision-language alignment under two complementary retrieval directions to characterize both discriminative labeling and representative item retrieval. We test the image-to-text classification, which treats each image as a query and ranks all candidate text labels by embedding similarity (cosine in the shared latent space); the predicted class is the top-1 label, and accuracy is computed over all images. This direction measures how well the model maps visual inputs to semantically correct category names without task-specific supervision on the evaluation set. The results are presented in Table~\ref{tab:test_c_c1_results}.
+
+
+\begin{table}[H]
+\centering
+\small
+\caption{Zero-shot classification (weighted F1) on 10\,k samples. Each model uses its own label vocabulary: the baseline uses dataset-native labels matching the FashionCLIP published protocol (``a photo of a \{label\}'', full 512-d cosine); GAP-CLIP uses its training labels. The baseline FMNIST wF1 of 74.04\% matches the published value of 0.74. Gain is relative to baseline: $(\text{GAP-CLIP} - \text{Baseline})/|\text{Baseline}|$.}
+\vspace{0.5em}
+\begin{tabular}{@{}lcccc@{}}
+\toprule
+\textbf{Dataset} & \textbf{Baseline (wF1)} & \textbf{GAP-CLIP (wF1)} & \textbf{Strategy} & \textbf{Gain} \\
+\midrule
+\multicolumn{5}{@{}l}{\emph{Single prompt (no strategy selection):}} \\
+\midrule
+Fashion-MNIST    & 74.04\% & 60.83\% & single\_prompt & $-$17.8\% \\
+KAGL Marqo       & 72.78\% & 43.49\% & single\_prompt & $-$40.2\% \\
+Internal dataset & 42.02\% & 44.75\% & single\_prompt & +6.5\%  \\
+\midrule
+\multicolumn{5}{@{}l}{\emph{GAP-CLIP best strategy:}} \\
+\midrule
+Fashion-MNIST    & 74.04\% & 61.65\% & blend\_hier\_full & $-$16.7\% \\
+KAGL Marqo       & 72.78\% & 68.65\% & blend\_hier\_full & $-$5.7\%  \\
+Internal dataset & 42.02\% & 79.31\% & hierarchy\_model  & +88.7\% \\
+\bottomrule
+\end{tabular}
+\label{tab:test_c_c1_results}
+\end{table}
+
+Under the single-prompt protocol, the baseline outperforms GAP-CLIP on the two external benchmarks (Fashion-MNIST and KAGL Marqo), reflecting the expected trade-off: GAP-CLIP's embedding geometry has been restructured to optimize attribute separation rather than raw zero-shot label matching. However, when GAP-CLIP leverages its structured subspaces via its best strategy, the gap narrows dramatically on KAGL Marqo (from $-$40.2\% to just $-$5.7\%) and inverts on the internal dataset, where GAP-CLIP reaches \textbf{79.31\% wF1}---a \textbf{+88.7\%} relative gain over the baseline's 42.02\%. This confirms that the strength of GAP-CLIP lies not in generic single-prompt zero-shot accuracy, but in its structured subspaces (Tests A and B: 100\% pass rate; Test D: 96.20\% decomposition consistency vs.\ 42.30\% baseline) and the dramatically higher separation scores reported above, which translate directly to better attribute-aware retrieval.
+
+\section{Discussion}
+
+The results presented across multiple datasets demonstrate that structuring the embedding space into semantically meaningful subspaces---color (16 dimensions) and category (64 dimensions)---offers a powerful compromise between interpretability and performance. Whereas standard CLIP models distribute semantic information across the full 512-dimensional space, our GAP-CLIP architecture explicitly isolates key fashion attributes, enabling both stronger disentanglement and refined control over the representation geometry. This is reflected in the substantial improvements in separation scores, which capture the degree to which the embedding space organizes items according to meaningful semantic factors. In particular, the gains observed for image-based separation (approaching +1{,}000\% relative improvement on some category and color predictions) highlight that specialized subspaces allow the model to extract much more discriminative visual cues than generic CLIP embeddings. \\
+
+However, this explicit specialization introduces an inherent tension: subspaces dramatically improve interpretability and concept-level isolation, yet the full 512-dimensional embedding remains superior for capturing fine-grained, high-capacity information. This trade-off is visible in the zero-shot evaluation (Table~\ref{tab:test_c_c1_results}): under a single-prompt protocol GAP-CLIP trails the FashionCLIP baseline on Fashion-MNIST ($-$17.8\%) and KAGL Marqo ($-$40.2\%), confirming that restructuring the geometry for attribute separation comes at a cost to generic label matching. Crucially, however, when GAP-CLIP exploits its structured subspaces via the \texttt{blend\_hier\_full} strategy---which blends hierarchy-subspace and full-embedding similarities---the gap narrows to just $-$5.7\% on KAGL Marqo and inverts entirely on the internal dataset, where GAP-CLIP achieves \textbf{79.31\% wF1} (+88.7\% relative gain). These results confirm that specialized subspaces and full embeddings are complementary rather than competitive: the blending strategy recovers much of the generic representational power while retaining the semantic transparency afforded by the subspaces. \\
+
+Another insight emerging from our experiments is the phenomenon of over-specialization. By tightly aligning specific dimensions with color and category information, the model may inadvertently discard broader semantic signals normally captured by a distributed representation. Some unused dimensions may in fact contain valuable residual semantics, and their forced deactivation could reduce generalization in domains where attribute boundaries are less clearly defined. Nevertheless, in fashion retrieval---where color and category are dominant attributes---our approach provides a highly interpretable and operationally advantageous embedding structure. \\
+
+Our results show that the standalone category model outperforms Fashion-CLIP on Fashion-MNIST, while the full GAP-CLIP model does not, due to fundamental differences in specialization and training objectives. The category model is a narrowly optimized classifier: its ResNet-based architecture and loss functions are designed exclusively to maximize category separability, producing clean, high-margin category embeddings that transfer well even to out-of-domain datasets like Fashion-MNIST. In contrast, GAP-CLIP is a multi-objective vision--language model that must simultaneously preserve CLIP semantics, encode color and category in prescribed subspaces, and generalize across diverse e-commerce tasks. This induces representational trade-offs: the category signal is compressed alongside other factors, and no task-specific head is trained for Fashion-MNIST, making zero-shot classification strictly harder. Combined with the strong domain shift between real RGB product photos and grayscale Fashion-MNIST silhouettes, the baseline Fashion-CLIP---despite weaker category structure---retains an advantage in global visual robustness, explaining the $-$16.7\% gap even under GAP-CLIP's best blending strategy. Notably, however, this gap shrinks substantially on the in-domain KAGL Marqo benchmark ($-$5.7\%) and reverses on the internal dataset (+88.7\%), suggesting that as domain alignment improves, GAP-CLIP's structured subspaces become increasingly beneficial. \\
+
+Finally, the semantic robustness experiments demonstrate that GAP-CLIP remains stable even under long, detailed, multi-attribute descriptions, whereas baseline CLIP significantly degrades. This resilience to textual verbosity suggests that explicitly encoded subspaces serve as semantic anchors, guiding the model even when irrelevant linguistic features are present.
+
+\newpage
+\section{Conclusion}
+
+GAP-CLIP introduces a principled architectural modification to CLIP, enabling fine-grained control over embedding semantics through dedicated subspaces. This design enhances interpretability, boosts disentanglement, and yields several-fold improvements in the separation of key fashion attributes such as color and category across diverse datasets. Zero-shot evaluation on 10\,000 samples confirms an expected trade-off: under a single-prompt protocol, GAP-CLIP trails the FashionCLIP baseline on generic benchmarks (Fashion-MNIST: $-$17.8\%; KAGL Marqo: $-$40.2\%), yet when it leverages its structured subspaces via a blending strategy the gap narrows dramatically (KAGL Marqo: $-$5.7\%) and inverts on the in-domain internal dataset (+88.7\%, reaching 79.31\% wF1). Combined with 100\% subspace-alignment pass rates (Tests A--B), 96.20\% decomposition consistency (Test~D), and strong resilience to textual verbosity, these results demonstrate that GAP-CLIP offers a compelling alternative to standard CLIP for fashion applications requiring semantic clarity, attribute-level control, and robust multi-attribute retrieval.
+
+
+\section{Limitations}
+
+Despite its advantages, the proposed model presents several limitations:
+\begin{itemize}
+    \item Dimensional specialization may lead to partial information loss, as unused or constrained dimensions could otherwise store relevant semantic nuances.
+    \item The approach requires pretraining specialized models, increasing the computational cost compared to a single end-to-end CLIP model.
+    \item Over-specialization may reduce performance in tasks where attributes are not cleanly separable or where broader semantic features are needed.
+\end{itemize}
+
+\section{Future Work}
+
+Several research directions naturally emerge from this work, such as extending attribute-specific subspaces to other fashion attributes such as style, texture, silhouette, or material, end-to-end joint training of the specialized subspaces and the full embedding model to optimize the trade-off between separation and generalization, adaptive subspace allocation, allowing the model to learn the optimal dimensional budget for each attribute.
+
+
+\newpage
+\bibliographystyle{plain}
+\begin{thebibliography}{99}
+
+\bibitem{marqo} Marqo Team. KAGL Dataset. 2023. 
+Link: https://huggingface.co/datasets/Marqo/KAGL
+
+\bibitem{mnist} Zalando Research. Fashion-MNIST Dataset. 2017.
+Link : https://www.kaggle.com/datasets/zalando-research/fashionmnist
+
+\bibitem{laion} Schuhmann, C., Beaumont, R., Vencu, R., et al.
+LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models.
+NeurIPS Datasets and Benchmarks, 2022.
+Link: https://arxiv.org/abs/2210.08402
+\textit{Model:} \texttt{laion/CLIP-ViT-B-32-laion2B-s34B-b79K}, an open-source CLIP ViT-B/32 trained on LAION-2B.
+
+\bibitem{clip} Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., et al.
+Learning Transferable Visual Models From Natural Language Supervision, 2021.
+Link: https://arxiv.org/abs/2103.00020
+
+\bibitem{liu2016deepfashion} Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.
+DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations.
+CVPR, 2016.
+
+\bibitem{hsiao2018learning} Hsiao, E., \& Grauman, K.
+Learning the Latent “Look”: Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images.
+ICCV, 2018.
+
+\bibitem{jia2021scaling} Jia, C., Yang, Y., Xia, X., et al.
+Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision.
+ICML, 2021.
+
+\bibitem{phang2022multimodal} Phang, J., Li, Y., Li, T., et al.
+Multimodal Pretraining Beyond Vision and Language.
+NeurIPS, 2022.
+
+\bibitem{ak2022attribute} Ak, K., et al.
+Attribute-based Clothing Retrieval: A Survey of Methods and Applications.
+International Journal of Computer Vision (IJCV), 2022.
+
+\bibitem{chen2020fashion} Chen, Y., Gu, X., et al.
+Fashion Compatibility Modeling via Contextuality and Compatibility Learning.
+CVPR, 2020.
+
+\bibitem{li2020learning} Li, T., Li, J., et al.
+Learning Fine-Grained Attributes in Fashion via Multi-Task Representation Learning.
+ECCV, 2020.
+
+\bibitem{li2022modular} Li, X., Chen, C., et al.
+Modular Representation Learning for Interpretable Medical Imaging.
+MICCAI, 2022.
+
+\bibitem{zhou2023geospatial} Zhou, Y., Nguyen, A., et al.
+Geospatial Vision–Language Pretraining with Modular Encoders.
+NeurIPS, 2023.
+
+\bibitem{baseline} Chia, P. J., et al.
+Contrastive Language and Vision Learning of General Fashion Concepts.
+arXiv, 2023.
+Link : https://arxiv.org/abs/2204.03972
+
+\bibitem{optuna} Akiba, T., Sano, S., Yanase, T., Ohta, T., \& Koyama, M.
+Optuna: A Next-generation Hyperparameter Optimization Framework.
+KDD, 2019.
+Link: https://arxiv.org/abs/1907.10902
+
+\bibitem{tsne}  Van der Maaten, L., \& Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(Nov)
+\end{thebibliography}
+
+\newpage
+\appendix
+\section{Annexes}
+
+\subsection{Main model color part heatmap}
+\label{heatmap}
+\begin{figure}[h!]
+    \centering
+    \begin{subfigure}[b]{0.45\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{heatmap_baseline.jpg}
+        \caption{Fashion clip baseline}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.5\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{heatmap.png}
+        \caption{GAP clip model}
+    \end{subfigure}
+    \caption{Heatmaps representing the similarity between the colors embeddings, using only the first 16 dimensions of the main model embeddings and the full baseline embeddings}
+\end{figure}
+
+\subsection{Main model t-SNE on test dataset}
+\label{tsne}
+\begin{figure}[h!]
+    \centering
+
+    \begin{subfigure}[b]{0.45\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{tsne_baseline.png}
+        \caption{Fashion clip baseline}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.45\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{tsne_model.png}
+        \caption{GAP clip model}
+    \end{subfigure}
+
+    \caption{t-SNE representing the clusters between the color embeddings, using only the first 16 dimensions of the main model embeddings and the full baseline embeddings}
+\end{figure}
+
+\begin{figure}[h!]
+    \centering
+
+    \begin{subfigure}[b]{0.45\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{tsne_hierarchy_baseline.png}
+        \caption{Fashion clip baseline}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.45\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{tsne_hierarchy_our.png}
+        \caption{GAP clip model}
+    \end{subfigure}
+
+    \caption{t-SNE representing the clusters between the hierarchy embeddings, using only the 64 dimensions of the main model embeddings and the full baseline embeddings}
+\end{figure}
+
+\newpage
+\subsection{Search Engine Examples}
+\label{search_engine}
+\begin{figure}[h!]
+    \centering
+    \begin{subfigure}[b]{0.8\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{gapclip_red_dress.png}
+        \caption{GAP-CLIP: ``red dress'' query}
+    \end{subfigure}
+
+    \vspace{0.5em}
+
+    \begin{subfigure}[b]{0.8\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{baseline_red_dress.png}
+        \caption{Baseline: ``red dress'' query}
+    \end{subfigure}
+
+    \caption{Search results for ``red dress''. GAP-CLIP (a) returns items that are consistently red dresses, while the baseline (b) shows more variability in both color and category.}
+\end{figure}
+
+\begin{figure}[h!]
+    \centering
+    \begin{subfigure}[b]{0.8\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{gapclip_blue_pant.png}
+        \caption{GAP-CLIP: ``blue pant'' query}
+    \end{subfigure}
+
+    \vspace{0.5em}
+
+    \begin{subfigure}[b]{0.8\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{baseline_blue_pant.png}
+        \caption{Baseline: ``blue pant'' query}
+    \end{subfigure}
+
+    \caption{Search results for ``blue pant''. GAP-CLIP (a) correctly retrieves blue pants, demonstrating strong attribute alignment. The baseline (b) shows less precise color and category matching.}
+\end{figure}
+
+\subsection{Hyperparameters importance with optuna}
+\label{optuna}
+\begin{figure}[H]
+    \centering
+    \includegraphics[width=0.6\textwidth]{optuna_param_importances.png}
+    \caption{Hyperparameters importance with optuna}
+\end{figure}
+
+
+\subsection{Main Model Training Curves}
+\label{training_curves}
+\begin{figure}[H]
+    \centering
+    \includegraphics[width=0.6\textwidth]{training_curves.png}
+    \caption{Main model training curves.}
+\end{figure}
+
+\subsection{Long descriptions examples}
+\label{long_desc}
+
+\noindent\textbf{description${_1}$:}
+\begin{quote}
+The specified item is a blue suit top made exclusively for Banana Republic Factory. It features a notch lapel, three quarter length sleeves, a two-button closure, four functional exterior pockets, a center back vent, and is fully lined. The style is a suit top, and the color is blue. \\
+Prediction for the gap clip model: shirt \\
+Prediction for the baseline model: jacket \\
+\end{quote}
+
+\noindent\textbf{description${_2}$:}
+\begin{quote}
+The specified item is a bodysuit. The style is a top with the fit and function of a bodysuit. The bodysuit is made of supersoft cotton jersey and features long sleeves with shirring at the shoulders, a round neckline with a pleat at the center, a decorative bow, a keyhole button and loop closure at the back, and an allover striped pattern. The bodysuit also has snaps at the inseam for easy dressing and diapering. \\
+Prediction for the gap clip model: shirt\\
+Prediction for the baseline model: sweater\\
+\end{quote}
+
+\noindent\textbf{description${_3}$:}
+\begin{quote}
+The specified item is a long-sleeved, soft double-knit slub jersey T-shirt in a brown color. The T-shirt features long sleeves with banded cuffs and a crewneck style. It is made from a soft, double-knit slub jersey material, which is designed for comfort and ease of movement. The T-shirt is hanging on a wooden hanger, ready for wear.\\
+Prediction for the gap clip model: t-shirt \\
+Prediction for the baseline model: sweater \\
+\end{quote}
+
+
+\noindent\textbf{description${_4}$:} The specified item is a silk dress with short sleeves in the color Lavender. It has a flowing A-line silhouette. \\
+Prediction for the gap clip model: Lavender \\
+Prediction for the baseline model: PastelPurple \\
+
+\noindent\textbf{description${_5}$:} 
+Description: The specified item is a casual t-shirt with long sleeves in the color Blue. It has a comfortable round neckline. \\
+Prediction for the gap clip model: Blue \\
+Prediction for the baseline model: RoyalBlue \\
+
+\noindent\textbf{description${_6}$:} 
+Description: The specified item is a fleece dress with long sleeves in the color Navy. It has a beautiful design.
+Prediction for the gap clip model: Navy \\
+Prediction for the baseline model: RoyalBlue \\
+
+\noindent\textbf{description${_7}$:} 
+Description: The specified item is a chiffon skirt with elastic waistband in the color Purple. It has a flowing midi-length cut. \\
+Prediction for the gap clip model: Purple \\
+Prediction for the baseline model: Amethyst \\
+
+
+\subsection{Confusion matrices}
+
+\begin{figure}[H]
+    \centering
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/local_text_color_confusion_matrix.png}
+        \caption{GAP-CLIP --- Text embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/local_baseline_text_color_confusion_matrix.png}
+        \caption{Baseline --- Text embeddings}
+    \end{subfigure}
+
+    \vspace{0.5em}
+
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/local_image_color_confusion_matrix.png}
+        \caption{GAP-CLIP --- Image embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/local_baseline_image_color_confusion_matrix.png}
+        \caption{Baseline --- Image embeddings}
+    \end{subfigure}
+    \caption{Color confusion matrices on the internal test dataset. GAP-CLIP (left) vs.\ Fashion-CLIP baseline (right) for text (top) and image (bottom) modalities.}
+    \label{fig:color_cm_local}
+\end{figure}
+
+\begin{figure}[H]
+    \centering
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/kaggle_text_color_confusion_matrix.png}
+        \caption{GAP-CLIP --- Text embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/kaggle_baseline_text_color_confusion_matrix.png}
+        \caption{Baseline --- Text embeddings}
+    \end{subfigure}
+
+    \vspace{0.5em}
+
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/kaggle_image_color_confusion_matrix.png}
+        \caption{GAP-CLIP --- Image embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_color/kaggle_baseline_image_color_confusion_matrix.png}
+        \caption{Baseline --- Image embeddings}
+    \end{subfigure}
+    \caption{Color confusion matrices on the KAGL Marqo dataset. GAP-CLIP (left) vs.\ Fashion-CLIP baseline (right) for text (top) and image (bottom) modalities.}
+    \label{fig:color_cm_kaggle}
+\end{figure}
+
+\begin{figure}[H]
+    \centering
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/gap_clip_internal_text_hierarchy_confusion_matrix.png}
+        \caption{GAP-CLIP --- Text embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/baseline_internal_text_hierarchy_confusion_matrix.png}
+        \caption{Baseline --- Text embeddings}
+    \end{subfigure}
+
+    \vspace{0.5em}
+
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/gap_clip_internal_image_hierarchy_confusion_matrix.png}
+        \caption{GAP-CLIP --- Image embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/baseline_internal_image_hierarchy_confusion_matrix.png}
+        \caption{Baseline --- Image embeddings}
+    \end{subfigure}
+    \caption{Category confusion matrices on the internal test dataset. GAP-CLIP (left) vs.\ Fashion-CLIP baseline (right) for text (top) and image (bottom) modalities.}
+    \label{fig:cat_cm_local}
+\end{figure}
+
+\begin{figure}[H]
+    \centering
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/gap_clip_kagl_marqo_text_hierarchy_confusion_matrix.png}
+        \caption{GAP-CLIP — Text embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/baseline_kagl_marqo_text_hierarchy_confusion_matrix.png}
+        \caption{Baseline — Text embeddings}
+    \end{subfigure}
+
+    \vspace{0.5em}
+
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/gap_clip_kagl_marqo_image_hierarchy_confusion_matrix.png}
+        \caption{GAP-CLIP — Image embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/baseline_kagl_marqo_image_hierarchy_confusion_matrix.png}
+        \caption{Baseline — Image embeddings}
+    \end{subfigure}
+    \caption{Category confusion matrices on the KAGL Marqo dataset. GAP-CLIP (left) vs.\ Fashion-CLIP baseline (right) for text (top) and image (bottom) modalities.}
+    \label{fig:cat_cm_kaggle}
+\end{figure}
+
+\begin{figure}[H]
+    \centering
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/gap_clip_text_hierarchy_confusion_matrix.png}
+        \caption{GAP-CLIP — Text embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/baseline_text_hierarchy_confusion_matrix.png}
+        \caption{Baseline — Text embeddings}
+    \end{subfigure}
+
+    \vspace{0.5em}
+
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/gap_clip_image_hierarchy_confusion_matrix.png}
+        \caption{GAP-CLIP — Image embeddings}
+    \end{subfigure}
+    \hfill
+    \begin{subfigure}[b]{0.48\textwidth}
+        \centering
+        \includegraphics[width=\textwidth]{confusion_matrices/cm_hierarchy/baseline_image_hierarchy_confusion_matrix.png}
+        \caption{Baseline — Image embeddings}
+    \end{subfigure}
+    \caption{Category confusion matrices on Fashion-MNIST. GAP-CLIP (left) vs.\ Fashion-CLIP baseline (right) for text (top) and image (bottom) modalities. Despite the domain shift from RGB to grayscale, GAP-CLIP maintains structured category predictions.}
+    \label{fig:cat_cm_fashion}
+\end{figure}
+
+
+\end{document}