Masoumeh Zareapoor
@shanghai jiao tong university
Computer science
Senior researcher
Masoumeh received her Ph.D. in computer science from Jamia Hamdard University, New Delhi, India. She has been a researcher at East China Normal University and Shanghai Jiao Tong University, Shanghai, China. Additionally, she worked as a visiting researcher at ETS in Montreal, to conduct cutting-edge research on deep learning models that are specialized for image geo-localization. Her research activities mainly focus on machine learning, computer vision, and image processing.
RESEARCH, TEACHING, or OTHER INTERESTS
Artificial Intelligence, Computer Vision and Pattern Recognition, Computers in Earth Sciences
Scopus Publications
Scholar Citations
Scholar h-index
Scholar i10-index
Scopus Publications
Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, and Michael Felsberg
Association for the Advancement of Artificial Intelligence (AAAI)
The dot product self-attention (DPSA) is a fundamental component of transformers. However, scaling them to long sequences, like documents or high-resolution images, becomes prohibitively expensive due to the quadratic time and memory complexities arising from the softmax operation. Kernel methods are employed to simplify computations by approximating softmax but often lead to performance drops compared to softmax attention. We propose SeTformer, a novel transformer where DPSA is purely replaced by Self-optimal Transport (SeT) for achieving better performance and computational efficiency. SeT is based on two essential softmax properties: maintaining a non-negative attention matrix and using a nonlinear reweighting mechanism to emphasize important tokens in input sequences. By introducing a kernel cost function for optimal transport, SeTformer effectively satisfies these properties. In particular, with small and base-sized models, SeTformer achieves impressive top-1 accuracies of 84.7% and 86.2% on ImageNet-1K. In object detection, SeTformer-base outperforms the FocalNet counterpart by +2.2 mAP, using 38% fewer parameters and 29% fewer FLOPs. In semantic segmentation, our base-size model surpasses NAT by +3.5 mIoU with 33% fewer parameters. SeTformer also achieves state-of-the-art results in language modeling on the GLUE benchmark. These findings highlight SeTformer applicability for vision and language tasks.
Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Xuelong Li, and Yue Lu
Elsevier BV
Masoumeh Zareapoor, Pourya Shamsolmoali, and Fateme Vesaghati
IEEE
Sparse Mixture-of-Experts (MoE) architectures provide the distinct benefit of substantially expanding the model’s parameter space without proportionally increasing the computational load on individual input tokens or samples. However, the efficacy of these models heavily depends on the routing strategy used to assign tokens to experts. Poor routing can lead to under-trained or overly specialized experts, diminishing the overall model performance. Previous approaches have relied on the Topk router, where each token is assigned to a subset of experts. In this paper, we propose a routing mechanism that replaces the Topk router with regularized optimal transport, leveraging the Sinkhorn algorithm to optimize token-expert matching. We conducted a comprehensive evaluation comparing the pre-training efficiency of our model, using computational resources equivalent to those employed in the GShard and Switch Transformers gating mechanisms. The results demonstrate that our model expedites training convergence, achieving a speedup of over 2× compared to these baseline models. Moreover, under the same computational constraints, our model exhibits superior performance across eleven tasks from the GLUE and SuperGLUE benchmarks. We show that our model contributes to the optimization of token-expert matching in sparsely-activated MoE models, offering substantial gains in both training efficiency and task performance.
Pourya Shamsolmoali, Masoumeh Zareapoor, Swagatam Das, Eric Granger, and Salvador García
Institute of Electrical and Electronics Engineers (IEEE)
Capsule networks (CapsNets) aim to parse images into a hierarchy of objects, parts, and their relationships using a two-step process involving part-whole transformation and hierarchical component routing. However, this hierarchical relationship modeling is computationally expensive, which has limited the wider use of CapsNet despite its potential advantages. The current state of CapsNet models primarily focuses on comparing their performance with capsule baselines, falling short of achieving the same level of proficiency as deep convolutional neural network (CNN) variants in intricate tasks. To address this limitation, we present an efficient approach for learning capsules that surpasses canonical baseline models and even demonstrates superior performance compared with high-performing convolution models. Our contribution can be outlined in two aspects: first, we introduce a group of subcapsules onto which an input vector is projected. Subsequently, we present the hybrid Gromov-Wasserstein (HGW) framework, which initially quantifies the dissimilarity between the input and the components modeled by the subcapsules, followed by determining their alignment degree through optimal transport (OT). This innovative mechanism capitalizes on new insights into defining alignment between the input and subcapsules, based on the similarity of their respective component distributions. This approach enhances CapsNets' capacity to learn from intricate, high-dimensional data while retaining their interpretability and hierarchical structure. Our proposed model offers two distinct advantages: 1) its lightweight nature facilitates the application of capsules to more intricate vision tasks, including object detection; and 2) it outperforms baseline approaches in these demanding tasks. Our empirical findings illustrate that HGW capsules (HGWCapsules) exhibit enhanced robustness against affine transformations, scale effectively to larger datasets, and surpass CNN and CapsNet models across various vision tasks.
Amit Kumar Srivastava, Shishir Kumar, and Masoumeh Zareapoor
Springer Science and Business Media LLC
Jianmei Wang, Masoumeh Zareapoor, Yeh-Cheng Chen, Pourya Shamsolmoali, and Jinwen Xie
Emerald
PurposeThe purpose of the study is threefold: first, to identify what factors influence mobile users' willingness of news learning and sharing, second, to find out whether users' learning in the news platforms will affect their sharing behavior and third, to access the impact of sharing intention on actual sharing behavior on the mobile platform.Design/methodology/approachThis study proposes an influence mechanism model for examining the relationship among the factors, news learning and news sharing. The proposed mechanism includes factors at three levels: personal, interpersonal and social level. To achieve this, researchers collected data from 474 mobile news users in China to test the hypotheses. The tools SPSS 26.0 and AMOS 23.0 were used to analysis the reliability, validity, model fits and structural equation modeling (SEM), respectively.FindingsThe findings indicate that news learning on the mobile platforms is affected by self-efficacy and self-enhancement. And news sharing intention is influenced by self-efficacy, interpersonal trust, interpersonal reciprocity, online community identity and social norms positively. News sharing intention has a significant effect on news sharing behavior, but news learning has an insignificant relationship with new sharing.Originality/valueThis study provides practical guidelines for mobile platform operators and news media managers by explicating the various factors of users' engagement on the news platforms. This paper also enriches the literature of news learning and news sharing on mobile by the integration of two theories: the social ecology theory and the interpersonal behavior theory.
Pourya Shamsolmoali, Masoumeh Zareapoor, Swagatam Das, Salvador Garcia, Eric Granger, and Jie Yang
Institute of Electrical and Electronics Engineers (IEEE)
Image-to-image (I2I) translation has become a key asset for generative adversarial networks. Convolutional neural networks (CNNs), despite having a significant performance, are not able to capture the spatial relationships among different parts of an object and, thus, do not qualify as the ideal representative model for image translation tasks. As a remedy to this problem, capsule networks have been proposed to represent patterns for a visual object in such a way that preserves hierarchical spatial relationships. The training of capsules is constrained by learning all pairwise relationships between capsules of consecutive layers. This design would be prohibitively expensive both in time and memory. In this article, we present a new framework for capsule networks to provide a full description of the input components at various levels of semantics, which can successfully be applied to the generator-discriminator architectures without incurring computational overhead compared to the CNNs. To successfully apply the proposed capsules in the generative adversarial network, we put forth a novel Gromov–Wasserstein (GW) distance as a differentiable loss function that compares the dissimilarity between two distributions and then guides the learned distribution toward target properties, using optimal transport (OT) discrepancy. The proposed method—which is called generative equivariant network (GEN)—is an alternative architecture for GANs with equivariance capsule layers. The proposed model is evaluated through a comprehensive set of experiments on I2I translation and image generation tasks and compared with several state-of-the-art models. Results indicate that there is a principled connection between generative and capsule models that allows extracting discriminant and invariant information from image data
Pourya Shamsolmoali, Masoumeh Zareapoor, and Eric Granger
IEEE
Image inpainting aims to generate realistic content for missing regions of an image. Existing methods often struggle to produce visually coherent content for missing regions of an image, which results in blurry or distorted structures around the damaged areas. These methods rely on surrounding texture information and have difficulty in generating content that harmonizes well with the broader context of the image. To address this limitation, we propose a novel model that generates plausible content for missing regions while ensuring that the generated content is consistent with the overall context of the original image. In particular, we introduce a novel context-adaptive transformer for image inpainting (TransInpaint) that relies on the visible content and the position of the missing regions. Additionally, we design a texture enhancement network that combines skip features from the encoder with the coarse features produced by the generator, yielding a more comprehensive and robust representation of image content. Based on extensive evaluations on challenging datasets, our proposed TransInpaint outperforms the cutting-edge generative models for image inpainting in terms of quality, textures, and structures.
Pourya Shamsolmoali, Masoumeh Zareapoor, and Eric Granger
IEEE
Given the recent advances with image-generating algorithms, deep image completion methods have made significant progress. However, state-of-art methods typically provide poor cross-scene generalization, and generated masked areas often contain blurry artifacts. Predictive filtering is a method for restoring images, which predicts the most effective kernels based on the input scene. Motivated by this approach, we address image completion as a filtering problem. Deep feature-level semantic filtering is introduced to fill in missing information, while preserving local structure and generating visually realistic content. In particular, a Dual-path Cooperative Filtering (DCF) model is proposed, where one path predicts dynamic kernels, and the other path extracts multi-level features by using Fast Fourier Convolution to yield semantically coherent reconstructions. Experiments on three challenging image completion datasets show that our proposed DCF outperforms state-of-art methods.
Pourya Shamsolmoali and Masoumeh Zareapoor
IEEE
This paper focuses on an accurate and fast interpolation approach for image transformation employed in the design of CNN architectures. Standard Spatial Transformer Networks (STNs) use bilinear or linear interpolation as their interpolation, with unrealistic assumptions about the underlying data distributions, which leads to poor performance under scale variations. Moreover, STNs do not preserve the norm of gradients in propagation due to their dependency on sparse neighboring pixels. To address this problem, a novel Entropy STN (ESTN) is proposed that interpolates on the data manifold distributions. In particular, random samples are generated for each pixel in association with the tangent space of the data manifold, and construct a linear approximation of their intensity values with an entropy regularizer to compute the transformer parameters. A simple yet effective technique is also proposed to normalize the non-zero values of the convolution operation, to fine-tune the layers for gradients' norm-regularization during training. Experiments on challenging benchmarks show that the proposed ESTN can improve predictive accuracy over a range of computer vision tasks, including image reconstruction, and classification, while reducing the computational cost.
Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Dacheng Tao, and Xuelong Li
Institute of Electrical and Electronics Engineers (IEEE)
Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables and these models use a non-linear function (generator) to map latent samples into the data space. On the other hand, the non-linearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. This weak projection, however, can be addressed by a Riemannian metric, and we show that geodesics computation and accurate interpolations between data samples on the Riemannian manifold can substantially improve the performance of deep generative models. In this paper, a Variational spatial-Transformer AutoEncoder (VTAE) is proposed to minimize geodesics on a Riemannian manifold and improve representation learning. In particular, we carefully design the variational autoencoder with an encoded spatial-Transformer to explicitly expand the latent variable model to data on a Riemannian manifold, and obtain global context modelling. Moreover, to have smooth and plausible interpolations while traversing between two different objects’ latent representations, we propose a geodesic interpolation network different from the existing models that use linear interpolation with inferior performance. Experiments on benchmarks show that our proposed model can improve predictive accuracy and versatility over a range of computer vision tasks, including image interpolations, and reconstructions.
Pourya Shamsolmoali, Masoumeh Zareapoor, Jie Yang, Eric Granger, and Huiyu Zhou
IEEE
Skin lesion detection in dermoscopic images is essential in the accurate and early diagnosis of skin cancer by a computerized apparatus. Current skin lesion segmentation approaches show poor performance in challenging circumstances such as indistinct lesion boundaries, low contrast between the lesion and the surrounding area, or heterogeneous background that causes over/under segmentation of the skin lesion. To accurately recognize the lesion from the neighboring regions, we propose a dilated scale-wise feature fusion network based on convolution factorization. Our network is designed to simultaneously extract features at different scales which are systematically fused for better detection. The proposed model has satisfactory accuracy and efficiency. Various experiments for lesion segmentation are performed along with comparisons with the state-of-the-art models. Our proposed model consistently showcases state-of-the-art results.
Pourya Shamsolmoali, Masoumeh Zareapoor, Jie Yang, Eric Granger, and Jocelyn Chanussot
IEEE
Small-object detection is a challenging problem. In the last few years, the convolution neural networks methods have been achieved considerable progress. However, the current detectors struggle with effective features extraction for small-scale objects. To address this challenge, we propose image pyramid single-shot detector (IPSSD). In IPSSD, single-shot detector is adopted combined with an image pyramid network to extract semantically strong features for generating candidate regions. The proposed network can enhance the small-scale features from a feature pyramid network. We evaluated the performance of the proposed model on two public datasets and the results show the superior performance of our model compared to the other state-of-the-art object detectors.
Pourya Shamsolmoali, Jocelyn Chanussot, Masoumeh Zareapoor, Huiyu Zhou, and Jie Yang
Institute of Electrical and Electronics Engineers (IEEE)
Object detection is a challenging task in remote sensing because objects only occupy a few pixels in the images, and the models are required to simultaneously learn object locations and detection. Even though the established approaches well perform for the objects of regular sizes, they achieve weak performance when analyzing small ones or getting stuck in the local minima (e.g., false object parts). Two possible issues stand in their way. First, the existing methods struggle to perform stably on the detection of small objects because of the complicated background. Second, most of the standard methods used handcrafted features and do not work well on the detection of objects parts that are missing. We here address the above issues and propose a new architecture with a multipatch feature pyramid network (MPFP-Net). Different from the current models that, during training, only pursue the most discriminative patches, in MPFP-Net, the patches are divided into class-affiliated subsets, in which the patches are related, and based on the primary loss function, a sequence of smooth loss functions is determined for the subsets to improve the model for collecting small object parts. To enhance the feature representation for patch selection, we introduce an effective method to regularize the residual values and make the fusion transition layers strictly norm-preserving. The network contains bottom-up and crosswise connections to fuse the features of different scales to achieve better accuracy compared to several state-of-the-art object detection models. Also, the developed architecture is more efficient than the baselines.
Pourya Shamsolmoali, Masoumeh Zareapoor, Jocelyn Chanussot, Huiyu Zhou, and Jie Yang
Institute of Electrical and Electronics Engineers (IEEE)
Detection of objects is extremely important in various aerial vision-based applications. Over the last few years, the methods based on convolution neural networks (CNNs) have made substantial progress. However, because of the large variety of object scales, densities, and arbitrary orientations, the current detectors struggle with the extraction of semantically strong features for small-scale objects by a predefined convolution kernel. To address this problem, we propose the rotation equivariant feature image pyramid network (REFIPN), an image pyramid network based on rotation equivariance convolution. The proposed model adopts single-shot detector in parallel with a lightweight image pyramid module (LIPM) to extract representative features and generate regions of interest in an optimization approach. The proposed network extracts feature in a wide range of scales and orientations by using novel convolution filters. These features are used to generate vector fields and determine the weight and angle of the highest-scoring orientation for all spatial locations on an image. By this approach, the performance for small-sized object detection is enhanced without sacrificing the performance for large-sized object detection. The performance of the proposed model is validated on two commonly used aerial benchmarks and the results show our proposed model can achieve state-of-the-art performance with satisfactory efficiency.
Lu Wang, Masoumeh Zareapoor, Jie Yang, and Zhonglong Zheng
Institute of Electrical and Electronics Engineers (IEEE)
In recent years, cross-modal hashing (CMH) has attracted considerable attention due to its ability to learn across different modalities and its high efficiency for similarity retrieval applications. This procedure is computationally inexpensive when dealing with large-scale multi-modalities datasets. However, they do not form the ideal representative model to fully exploit multi-modal data’s underlying properties despite their successful performance. We identify that: (i) most CMH models in their current forms transform the real data points into discrete compact binary codes, which can limit their ability to prevent the loss of important information and thereby produce suboptimal results. (ii) the discrete-binary constraint model is hard to implement, and relaxing the binary constraints is a common property in most existing methods, which often leads to significant quantization errors. (iii) handling the CMH in a symmetry domain leads to a complex and inefficient optimization problem. This paper addresses the above challenges and proposes a novel Asymmetric Correlation Quantization Hashing (ACQH) method. ACQH learns a projection matrix for each heterogeneous modality to map the data point into a low-dimensional semantic space and constructs a compositional quantization to generate hash codes, using the pairwise semantic similarity preservation and the pointwise label regression. As a specific instantiation of our model, we use discrete iterative optimization to obtain the unified hash codes across different modalities. Extensive experiments show that ACQH outperforms state-of-the-art methods on several diverse datasets.
Pourya Shamsolmoali, Masoumeh Zareapoor, Linlin Shen, Abdul Hamid Sadka, and Jie Yang
Elsevier BV
Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Huiyu Zhou, Ruili Wang, M. Emre Celebi, and Jie Yang
Elsevier BV
Masoumeh Zareapoor and Jie Yang
Association for Computing Machinery (ACM)
Image-to-Image translation aims to learn an image from a source domain to a target domain. However, there are three main challenges, such as lack of paired datasets, multimodality, and diversity, that are associated with these problems and need to be dealt with. Convolutional neural networks (CNNs), despite of having great performance in many computer vision tasks, they fail to detect the hierarchy of spatial relationships between different parts of an object and thus do not form the ideal representative model we look for. This article presents a new variation of generative models that aims to remedy this problem. We use a trainable transformer, which explicitly allows the spatial manipulation of data within training. This differentiable module can be augmented into the convolutional layers in the generative model, and it allows to freely alter the generated distributions for image-to-image translation. To reap the benefits of proposed module into generative model, our architecture incorporates a new loss function to facilitate an effective end-to-end generative learning for image-to-image translation. The proposed model is evaluated through comprehensive experiments on image synthesizing and image-to-image translation, along with comparisons with several state-of-the-art algorithms.
Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Ruili Wang, and Jie Yang
Institute of Electrical and Electronics Engineers (IEEE)
Road extraction in remote sensing images is of great importance for a wide range of applications. Because of the complex background, and high density, most of the existing methods fail to accurately extract a road network that appears correct and complete. Moreover, they suffer from either insufficient training data or high costs of manual annotation. To address these problems, we introduce a new model to apply structured domain adaption for synthetic image generation and road segmentation. We incorporate a feature pyramid (FP) network into generative adversarial networks to minimize the difference between the source and target domains. A generator is learned to produce quality synthetic images, and the discriminator attempts to distinguish them. We also propose a FP network that improves the performance of the proposed model by extracting effective features from all the layers of the network for describing different scales’ objects. Indeed, a novel scale-wise architecture is introduced to learn from the multilevel feature maps and improve the semantics of the features. For optimization, the model is trained by a joint reconstruction loss function, which minimizes the difference between the fake images and the real ones. A wide range of experiments on three data sets prove the superior performance of the proposed approach in terms of accuracy and efficiency. In particular, our model achieves state-of-the-art 78.86 IOU on the Massachusetts data set with 14.89M parameters and 86.78B FLOPs, with $4\\times $ fewer FLOPs but higher accuracy (+3.47% IOU) than the top performer among state-of-the-art approaches used in the evaluation.
Lu Wang, Jie Yang, Masoumeh Zareapoor, and Zhonglong Zheng
Elsevier BV
Masoumeh Zareapoor, Pourya Shamsolmoali, and Jie Yang
Elsevier BV
Donghao Shen, Masoumeh Zareapoor, and Jie Yang
Elsevier BV
Donghao Shen, Masoumeh Zareapoor, and Jie Yang
Elsevier BV
Hafiz Tayyab Mustafa, Jie Yang, Hamza Mustafa, and Masoumeh Zareapoor
Elsevier BV
RECENT SCHOLAR PUBLICATIONS
MOST CITED SCHOLAR PUBLICATIONS
Publications
M Zareapoor, P Shamsolmoali, F Vesaghati, An Efficient Sparse Mixture of Experts. International Joint Conference on Neural Networks (IJCNN), 2024.
M. Zareapoor, P. Shamsolmoali, Y. Lu, E. Granger, J. Yang, Mapping the Invisible: Object Detection in Remote Sensing Imagery via Cost-Regularized Optimal Transport, ISPRS Journal of Photogrammetry and Remote Sensing, 2024.