Efficient Parallel Transformers of Mixture-of-Experts (EPT-MoE) coupled with Spatial Feed Forward Neural Networks (SFFN) to enhance the ability of parallel Transformer models with Mixture-of-Experts layers for graph learning of 3D skeleton-data hand gesture recognition.
The Mixture-of-Experts (MoE) is a widely known deep neural architecture where an ensemble of specialized sub-models (a group of experts) optimizes the overall performance with a constant computational cost. Especially with the rise of Mixture-of-Experts with Mixtral-8x7B Transformers, MoE architectures have gained popularity in Large Language Modeling (LLM) and Computer Vision. In this paper, we propose the Efficient Parallel Transformers of Mixture-of-Experts (EPT-MoE) coupled with Spatial Feed Forward Neural Networks (SFFN) to enhance the ability of parallel Transformer models with Mixture-of-Experts layers for graph learning of 3D skeleton-data hand gesture recognition. Nowadays, 3D hand gesture recognition is an attractive field of research in human-computer interaction, VR/AR and pattern recognition. For this purpose, our proposed EPT-MoE model decouples the spatial and temporal graph learning of 3D hand gestures by integrating mixture-of-experts layers into parallel Transformer m…
The Mixture-of-Experts (MoE) is a widely known deep neural architecture where an ensemble of specialized sub-models (a group of experts) optimizes the overall performance with a constant computational cost. We conduct extensive experiments for scaling studies and evaluation performance overall several variants of EPT-MoE model family to show the advantages of combining parallel transformer encoders with multiple layers of Mixture-of-Experts into one model transformer for 3D hand gesture recognition.
https://avestia.com/EECSS2024_Proceedings/files/paper/MVML/MVML_105.pdf
https://hal-lara.archives-ouvertes.fr/CESI/hal-04711525v1
Accelerating Training and Recognition Accuracy of EPT-MoE Models for 3D Hand Gesture Recognition
Abstract - The Mixture-of-Experts (MoE) is a widely known deep neural architecture where an ensemble of specialized sub-models (a
group of experts) optimizes the overall performance with a constant computational cost. Especially with the rise of Mixture-of-Experts
with Mixtral-8x7B Transformers, MoE architectures have gained popularity in Large Language Modeling (LLM) and Computer Vision.
In this paper, we propose the Efficient Parallel Transformers of Mixture-of-Experts (EPT-MoE) coupled with Spatial Feed Forward
Neural Networks (SFFN) to enhance the ability of parallel Transformer models with Mixture-of-Experts layers for graph learning of 3D
skeleton-data hand gesture recognition. Nowadays, 3D hand gesture recognition is an attractive field of research in human-computer
interaction, VR/AR and pattern recognition. For this purpose, our proposed EPT-MoE model decouples the spatial and temporal graph
learning of 3D hand gestures by integrating mixture-of-experts layers into parallel Tr…
