Search

Crowd counting aims to estimate the number and distribution of the population in crowded places, which is an important research direction in object counting. It is widely used in public place management, crowd behavior analysis, and other scenarios, showing its robust practicality. In recent years, crowd-counting technology has been developing rapidly. However, in highly crowded and noisy scenes, the counting effect of most models is still seriously affected by the distortion of view angle, dense occlusion, and inconsistent crowd distribution. Perspective distortion causes crowds to appear in different sizes and shapes in the image, and dense occlusion and inconsistent crowd distributions result in parts of the crowd not being captured completely. This ultimately results in the imperfect capture of spatial information in the model. To solve such problems, we propose a strip pooling combined attention (SPCANet) network model based on normed-deformable convolution (NDConv). We model long-distance dependencies more efficiently by introducing strip pooling. In contrast to traditional square kernel pooling, strip pooling uses long and narrow kernels (1×N or N×1) to deal with dense crowds, mutual occlusion, and overlap. Efficient channel attention (ECA), a mechanism for learning channel attention using a local cross-channel interaction strategy, is also introduced in SPCANet. This module generates channel attention through a fast 1D convolution to reduce model complexity while improving performance as much as possible. Four mainstream datasets, Shanghai Tech Part A, Shanghai Tech Part B, UCF-QNRF, and UCF CC 50, were utilized in extensive experiments, and mean absolute error (MAE) exceeds the baseline, which is 60.9, 7.3, 90.8, and 161.1, validating the effectiveness of SPCANet. Meanwhile, mean squared error (MSE) decreases by 5.7% on average over the four datasets, and the robustness is greatly improved. [ABSTRACT FROM AUTHOR]

Crowd counting aims to estimate the number and distribution of the population in crowded places, which is an important research direction in object counting. It is widely used in public place management, crowd behavior analysis, and other scenarios, showing its robust practicality. In recent years, crowd-counting technology has been developing rapidly. However, in highly crowded and noisy scenes, the counting effect of most models is still seriously affected by the distortion of view angle, dense occlusion, and inconsistent crowd distribution. Perspective distortion causes crowds to appear in different sizes and shapes in the image, and dense occlusion and inconsistent crowd distributions result in parts of the crowd not being captured completely. This ultimately results in the imperfect capture of spatial information in the model. To solve such problems, we propose a strip pooling combined attention (SPCANet) network model based on normed-deformable convolution (NDConv). We model long-distance dependencies more efficiently by introducing strip pooling. In contrast to traditional square kernel pooling, strip pooling uses long and narrow kernels (1×N or N×1) to deal with dense crowds, mutual occlusion, and overlap. Efficient channel attention (ECA), a mechanism for learning channel attention using a local cross-channel interaction strategy, is also introduced in SPCANet. This module generates channel attention through a fast 1D convolution to reduce model complexity while improving performance as much as possible. Four mainstream datasets, Shanghai Tech Part A, Shanghai Tech Part B, UCF-QNRF, and UCF CC 50, were utilized in extensive experiments, and mean absolute error (MAE) exceeds the baseline, which is 60.9, 7.3, 90.8, and 161.1, validating the effectiveness of SPCANet. Meanwhile, mean squared error (MSE) decreases by 5.7% on average over the four datasets, and the robustness is greatly improved.

Recently, with the growing popularity of mobile devices as well as video sharing platforms (e.g., YouTube, Facebook, TikTok, and Twitch), User-Generated Content (UGC) videos have become increasingly common and now account for a large portion of multimedia traffic on the internet. Unlike professionally generated videos produced by filmmakers and videographers, typically, UGC videos contain multiple authentic distortions, generally introduced during capture and processing by naive users. Quality prediction of UGC videos is of paramount importance to optimize and monitor their processing in hosting platforms, such as their coding, transcoding, and streaming. However, blind quality prediction of UGC is quite challenging, because the degradations of UGC videos are unknown and very diverse, in addition to the unavailability of pristine reference. Therefore, in this article, we propose an accurate and efficient Blind Video Quality Assessment (BVQA) model for UGC videos, which we name 2BiVQA for double Bi-LSTM Video Quality Assessment. 2BiVQA metric consists of three main blocks, including a pre-trained Convolutional Neural Network to extract discriminative features from image patches, which are then fed into two Recurrent Neural Networks for spatial and temporal pooling. Specifically, we use two Bi-directional Long Short-term Memory networks, the first is used to capture short-range dependencies between image patches, while the second allows capturing long-range dependencies between frames to account for the temporal memory effect. Experimental results on recent large-scale UGC VQA datasets show that 2BiVQA achieves high performance at lower computational cost than most state-of-the-art VQA models. The source code of our 2BiVQA metric is made publicly available at https://github.com/atelili/2BiVQA. [ABSTRACT FROM AUTHOR]

Delineation of retinal vessels in fundus images is essential for detecting a range of eye disorders. An automated technique for vessel segmentation can assist clinicians and enhance the efficiency of the diagnostic process. Traditional methods fail to extract multiscale information, discard unnecessary information, and delineate thin vessels. In this paper, a novel residual U-Net architecture that incorporates multi-scale feature learning and effective attention is proposed to delineate the retinal vessels precisely. Since drop block regularization performs better than drop out in preventing overfitting, drop block was used in this study. A multi-scale feature learning module was added instead of a skip connection to learn multi-scale features. A novel effective attention block was proposed and integrated with the decoder block to obtain precise spatial and channel information. Experimental findings indicated that the proposed model exhibited outstanding performance in retinal vessel delineation. The sensitivities achieved for DRIVE, STARE, and CHASE_DB datasets were 0.8293, 0.8151 and 0.8084, respectively.

Hypertensive retinopathy (HR) results from the microvascular retinal changes triggered by hypertension, which is the most common leading cause of preventable blindness worldwide. Therefore, it is necessary to develop an automated system for HR detection and evaluation using retinal images. We aimed to propose an automated approach to identify and categorize the various degrees of HR severity. A new network called the spatial convolution module (SCM) combines cross-channel and spatial information, and the convolution operations extract helpful features. The present model is evaluated using publicly accessible datasets ODIR, INSPIREVR, and VICAVR. We applied the augmentation to artificially increase the dataset of 1200 fundus images. The different HR severity levels of normal, mild, moderate, severe, and malignant are finally classified with the reduced time when compared to the existing models because in the proposed model, convolutional layers run only once on the input fundus images, which leads to a speedup and reduces the processing time in detecting the abnormalities in the vascular structure. According to the findings, the improved SVM had the highest detection and classification accuracy rate in the vessel classification with an accuracy of 98.99% and completed the task in 160.4 s. The ten-fold classification achieved the highest accuracy of 98.99%, i.e., 0.27 higher than the five-fold classification accuracy and the improved KNN classifier achieved an accuracy of 98.72%. When computation efficiency is a priority, the proposed model's ability to quickly recognize different HR severity levels is significant. [ABSTRACT FROM AUTHOR]

Transformers have demonstrated significant promise for computer vision tasks. Particularly noteworthy is SwinUNETR, a model that employs vision transformers, which has made remarkable advancements in improving the process of segmenting medical images. Nevertheless, the efficacy of training process of SwinUNETR has been constrained by an extended training duration, a limitation primarily attributable to the integration of the attention mechanism within the architecture. In this article, to address this limitation, we introduce a novel framework, called the MetaSwin model. Drawing inspiration from the MetaFormer concept that uses other token mix operations, we propose a transformative modification by substituting attention-based components within SwinUNETR with a straightforward yet impactful spatial pooling operation. Additionally, we incorporate of Squeeze-and-Excitation (SE) blocks after each MetaSwin block of the encoder and into the decoder, which aims at segmentation performance. We evaluate our proposed MetaSwin model on two distinct medical datasets, namely BraTS 2023 and MICCAI 2015 BTCV, and conduct a comprehensive comparison with the two baselines, i.e., SwinUNETR and SwinUNETR+SE models. Our results emphasize the effectiveness of MetaSwin, showcasing its competitive edge against the baselines, utilizing a simple pooling operation and efficient SE blocks. MetaSwin's consistent and superior performance on the BTCV dataset, in comparison to SwinUNETR, is particularly significant. For instance, with a model size of 24, MetaSwin outperforms SwinUNETR's 76.58% Dice score using fewer parameters (15,407,384 vs 15,703,304) and a substantially reduced training time (300 vs 467 mins), achieving an improved Dice score of 79.12%. This research highlights the essential contribution of a simplified transformer framework, incorporating basic elements such as pooling and SE blocks, thus emphasizing their potential to guide the progression of medical segmentation models, without relying on complex attentionbased mechanisms. [ABSTRACT FROM AUTHOR]

A material consequence of climate change is the intensification of extreme precipitation in most regions across the globe. The respective trend signal is already detectable at global to regional scales, but long-term variability still dominates local observational records, which are the basis for extreme precipitation risk assessment. Whether the frequency of extreme events is purely random or subject to a low-frequency internal variability forcing is therefore highly relevant for modelling the expected number of extreme events in a typical observational record. Based on millennial climate simulations, we show that long-term variability is largely random, with no clear indication of low-frequency decadal to multidecadal variability. Nevertheless, extreme precipitation events occur highly irregularly, with potential clustering (11% probability of five or more 100-year events in 250 years) or long disaster gaps with no events (8% probability for no 100-year events in 250 years). Even for decadal precipitation records, a complete absence of any tail events is not unlikely, as, for example, in typical 70-year observational or reanalysis data, the probability is almost 50%. This generally causes return levels – a key metric for infrastructure codes or insurance pricing – to be underestimated. We also evaluate the potential of employing information across neighbouring locations, which substantially improves the estimation of return levels by increasing the robustness against potential adverse effects of long-term internal variability. The irregular occurrence of events makes it challenging to estimate return periods for planning and for extreme event attribution.

Hypertensive retinopathy (HR) results from the microvascular retinal changes triggered by hypertension, which is the most common leading cause of preventable blindness worldwide. Therefore, it is necessary to develop an automated system for HR detection and evaluation using retinal images. We aimed to propose an automated approach to identify and categorize the various degrees of HR severity. A new network called the spatial convolution module (SCM) combines cross-channel and spatial information, and the convolution operations extract helpful features. The present model is evaluated using publicly accessible datasets ODIR, INSPIREVR, and VICAVR. We applied the augmentation to artificially increase the dataset of 1200 fundus images. The different HR severity levels of normal, mild, moderate, severe, and malignant are finally classified with the reduced time when compared to the existing models because in the proposed model, convolutional layers run only once on the input fundus images, which leads to a speedup and reduces the processing time in detecting the abnormalities in the vascular structure. According to the findings, the improved SVM had the highest detection and classification accuracy rate in the vessel classification with an accuracy of 98.99% and completed the task in 160.4 s. The ten-fold classification achieved the highest accuracy of 98.99%, i.e., 0.27 higher than the five-fold classification accuracy and the improved KNN classifier achieved an accuracy of 98.72%. When computation efficiency is a priority, the proposed model’s ability to quickly recognize different HR severity levels is significant.

Log data is an important clue to understanding the behaviour of a system at runtime, but the complexity of software systems in recent years has made the data that engineers need to analyse enormous and difficult to understand. While log-based anomaly detection methods based on deep learning have enabled highly accurate detection, the computational performance required to operate the models has become very high. In this study, we propose an anomaly detection method, SPClassifier, based on sparse features and the internal state of the model, and investigate the feasibility of anomaly detection that can be utilized in environments without computing resources such as GPUs. Benchmark with the latest deep learning models on the BGL dataset shows that the proposed method can achieve competitive accuracy with these methods and has a high level of anomaly detection performance even when the amount of training data is small.

Deep Learning based image quality assessment (IQA) has been shown to greatly improve the quality score prediction accuracy of images with single distortion. However, because these models lack generalizability and the accuracy of multidistortion-based image data is relatively low, designing reliable IQA systems is still an open issue. In this paper, we propose to introduce long-range dependencies between local artifacts and high-order spatial pooling into a convolutional neural network (CNN) model to improve the performance and generalizability of the full-reference IQA (FR-IQA). This long-range dependencies model is based on the hypothesis that local apparent artifacts can affect the overall image quality. The proposed network architecture adopts a non-local means algorithm to establish connections between all positions in the deep feature space and uses the Minkowski function to improve the non-linearity of the spatial pooling. Based on this architecture, a robust FR-IQA system has been constructed and evaluated on three well-known single-distortion-based IQA databases (LIVE, CSIQ, and TID2013) and a multidistortion-based IQA database (MDID). Experimental results demonstrate that, compared with the latest FR-IQA systems, the proposed long-range dependencies-boosted CNN-based FR-IQA system can achieve state-of-the-art performance. A comprehensive cross-database evaluation also shows that the proposed system is sufficiently generalized between different databases and multidistortion-based image data is more useful for training robust image quality metrics.

Diabetic Retinopathy (DR) is a micro vascular complication caused by long-term diabetes mellitus. Unidentified diabetic retinopathy leads to permanent blindness. Early identification of this disease requires frequent complex diagnostic procedure which is expensive and time consuming. In this article, we propose a composite deep neural network architecture with gated-attention mechanism for automated diagnosis of diabetic retinopathy. The feature descriptors obtained from multiple pre-trained deep Convolutional Neural Networks (CNNs) are used to represent color fundus retinal images. Spatial pooling methods are introduced to get the reduced versions of these representations without loosing much information. The proposed composite DNN learns independently from each of these reduced representations through different channels and contributes to improving the model generalization. In addition, model also includes gated attention blocks which allows the model to emphasize more on lesion portions of the retinal images while reduced attention to the non-lesion regions. Our experiments on APTOS-2019 Kaggle blindness detection challenge reveal that, the proposed approach leads to improved performance when compared to the existing best models. Our empirical studies also reveal that, the proposed approach leads to more generalised predictions with multi-modal representations when compared to those of uni-modal representations. The proposed composite deep neural network model recorded an accuracy of 82.54% (↑ 2%), and a Kappa score of 79 ( ↑ 9 points) for diabetic retinopathy severity level prediction. [ABSTRACT FROM AUTHOR]

Convolutional Neural Network (CNN) has brought significant improvements for various multimedia tasks. In contrast, image retrieval has not yet benefited as much since no training database is available. In this paper, we propose an unsupervised weighting scheme for pre-trained CNN models to adaptively emphasize image center. Different from the general preference for fully connected layers which represent abstract semantics, we aggregate the activations of convolutional layers on image patches to depict local patterns in details. It is an empirical observation that the target of searching is naturally the focus of an image. Thus we pooling the features with respect to their positions, since they innately maintain the geometric layout of an image. Experimental results on two benchmarks prove the effectiveness of our methods. [ABSTRACT FROM AUTHOR]

Deep hashing is the mainstream algorithm for large-scale cross-modal retrieval due to its high retrieval speed and low storage capacity, but the problem of reconstruction of modal semantic information is still very challenging. In order to further solve the problem of unsupervised cross-modal retrieval semantic reconstruction, we propose a novel deep semantic-preserving reconstruction hashing (DSPRH). The algorithm combines spatial and channel semantic information, and mines modal semantic information based on adaptive self-encoding and joint semantic reconstruction loss. The main contributions are as follows: (1) We introduce a new spatial pooling network module based on tensor regular-polymorphic decomposition theory to generate rank-1 tensor to capture high-order context semantics, which can assist the backbone network to capture important contextual modal semantic information. (2) Based on optimization perspective, we use global covariance pooling to capture channel semantic information and accelerate network convergence. In feature reconstruction layer, we use two bottlenecks auto-encoding to achieve visual-text modal interaction. (3) In metric learning, we design a new loss function to optimize model parameters, which can preserve the correlation between image modalities and text modalities. The DSPRH algorithm is tested on MIRFlickr-25K and NUS-WIDE. The experimental results show that DSPRH has achieved better performance on retrieval tasks.

Most of existing mesh quality assessment metrics consist of a similar two-stage computation process: first constructing a quality map by comparing the local regions between reference mesh and distorted mesh, then adopting a spatial pooling method to generate the overall quality score. In this paper, we propose a novel spatial pooling method for 3D mesh quality assessment based on percentile weighting strategy. We assign more weight to the severely distorted regions of the mesh at the pooling stage and extend the percentile weighting method by incorporating surface area at the pooling stage. Our analysis indicates that the percentile weighting method has a strong capability to emphasize the local regions with severe distortion of the mesh. We develop a mesh quality metric by pooling the local distances generated by the Tensor-based Perceptual Distance Measure metric with our spatial pooling method. We investigate the influence of the parameters of percentile weighting on the performance, and determine the optimal parameters and unified parameters through empirical tests on three publicly available databases. Experimental results demonstrate the effectiveness of our spatial pooling method and the superiority of our metric over state-of-the-art metrics. [ABSTRACT FROM AUTHOR]

The large number of object categories and many overlapping or closely neighboring objects in large-scale urban scenes pose great challenges in point cloud classification. Most works in deep learning have achieved a great success on regular input representations, but they are hard to be directly applied to classify point clouds due to the irregularity and inhomogeneity of the data. In this paper, a deep neural network with spatial pooling (DNNSP) is proposed to classify large-scale point clouds without rasterization. The DNNSP first obtains the point-based feature descriptors of all points in each point cluster. The distance minimum spanning tree-based pooling is then applied in the point feature representation to describe the spatial information among the points in the point clusters. The max pooling is next employed to aggregate the point-based features into the cluster-based features. To assure the DNNSP is invariant to the point permutation and sizes of the point clusters, the point-based feature representation is determined by the multilayer perception (MLP) and the weight sharing for each point is retained, which means that the weight of each point in the same layer is the same. In this way, the DNNSP can learn the features of points scaled from the entire regions to the centers of the point clusters, which makes the point cluster-based feature representations robust and discriminative. Finally, the cluster-based features are input to another MLP for point cloud classification. We have evaluated qualitatively and quantitatively the proposed method using several airborne laser scanning and terrestrial laser scanning point cloud data sets. The experimental results have demonstrated the effectiveness of our method in improving classification accuracy. [ABSTRACT FROM AUTHOR]

In computer graphics, various processing operations are applied to 3D triangle meshes and these processes often involve distortions, which affect the visual quality of surface geometry. In this context, perceptual quality assessment of 3D triangle meshes has become a crucial issue. In this paper, we propose a new objective quality metric for assessing the visual difference between a reference mesh and a corresponding distorted mesh. Our analysis indicates that the overall quality of a distorted mesh is sensitive to the distortion distribution. The proposed metric is based on a spatial pooling strategy and statistical descriptors of the distortion distribution. We generate a perceptual distortion map for vertices in the reference mesh while taking into account the visual masking effect of the human visual system. The proposed metric extracts statistical descriptors from the distortion map as the feature vector to represent the overall mesh quality. With the feature vector as input, we adopt a support vector regression model to predict the mesh quality score.We validate the performance of our method with three publicly available databases, and the comparison with state-of-the-art metrics demonstrates the superiority of our method. Experimental results show that our proposed method achieves a high correlation between objective assessment and subjective scores. [ABSTRACT FROM AUTHOR]

The spatial pooler in brain's neocortex models how the cortical neurons learn the feedforward connections and forms the cortical representation from various sensory information. To realize the spatial pooler's operation in hardware, we propose a new spatial-pooling memristor crossbar that converts the inputs from sensory organ into the sparse distributed representation (SDR) of cortical neurons, in this paper. The spatial pooling is composed of overlapping, inhibition, and learning steps. The proposed memristor crossbar can perform exactly the same functions of overlapping, inhibition, and learning with the spatial-pooling software algorithm. By converting the sensory information to SDR using the spatial-pooling memristor crossbar, we can have the following advantages for the cortical information processing: first, preserving the semantic similarity through the spatial pooling; second, changing synaptic weights continuously; third, keeping the sparsity of SDR as low as 2% like brain's neocortex; and fourth, improving the noise robustness. In this paper, these four properties of the spatial pooling in brain's neocortex have been simulated and verified in the memristor crossbar circuit. For verifying the spatial-pooling operation, the memristor crossbar has been tested for MNIST image recognition in this paper. The recognition rate is as high as 95% for the memristor crossbar with 4096 columns. [ABSTRACT FROM AUTHOR]

With the growing use of minimally invasive surgical procedures, endoscopic video archives are growing at a rapid pace. Efficient access to relevant content in such huge multimedia archives require compact and discriminative visual features for indexing and matching. In this paper, we present an effective method to represent images using salient convolutional features. Convolutional kernels from the first layer of a pre-trained convolutional neural network (CNN) are analyzed and clustered into multiple distinct groups, based on their sensitivity to colors and textures. Dominant features detected by each cluster are collected into a single, layout-preserving feature map using a spatial maximal activator pooling (SMAP) approach. A moving window based structured pooling method then captures spatial layout features and global shape information from the aggregated feature map to populate feature histograms. Finally, individual histograms for each cluster are combined into a single comprehensive feature histogram. Clustering convolutional feature space allow extraction of color and texture features of varying strengths. Further, the SMAP approach enable us to select dominant discriminative features. The proposed features are compact and capable of conveniently outperforming several existing features extraction approaches in retrieval and classification tasks on endoscopy images dataset. [ABSTRACT FROM AUTHOR]

Future extreme precipitation events are expected to be influenced by climate change; however, the robustness of this anthropogenically forced response in respect to projection uncertainty especially for sub-daily extremes is not fully understood. We analyze the impact of anthropogenic climate change on 3 hourly extreme precipitation with return periods ranging between 5 and 50 years over Europe using the RCA4 model ensemble simulations at 0.11° resolution. The robustness of the signals is examined based on a regionalized signal-to-noise (S2N) technique by taking the spatial pooling into account and the efficacy of the regionalization is tested by a sensitivity analysis. The results show an increasing signal in 3 hourly extreme precipitation over Europe for all seasons except summer for which a bipolar pattern (increase in the north and decrease in the south) is discerned. For the business-as-usual scenario RCP8.5, the regionalized winter 3 hourly extreme precipitation signals over 9 × 9 model grid cells are statistically significant in roughly 72%, 65%, 59% and 48% of the European area for 5, 15, 25 and 50 year return periods respectively, while 16%–21% of the area will experience significant changes in summer. The S2N values for 3 hourly extreme precipitation changes rise after the spatial pooling by about a factor of 1.4–1.7 for all seasons except summer when they decline by about a factor of 0.78. The results of sensitivity analysis reveal that the regionalization influence is sensitive—in order of decreasing importance—to season, precipitation time scale, precipitation intensity, emission scenario and model spatial resolution. The precipitation time scale is particularly important seasonally in summer and regionally in south Europe when/where short-duration convective precipitation is dominant.

The human visual system proves expert in discovering patterns in both global and local feature space. Can we design a similar way for unsupervised feature learning? In this paper, we propose a novel spatial pooling method within an unsupervised feature learning framework, named Rich and Robust Feature Pooling ( R 2 FP ), to better extract rich and robust representation from sparse feature maps learned from the raw data. Both local and global pooling strategies are further considered to instantiate such a method. The former selects the most representative features in the sub-region and summarizes the joint distribution of the selected features, while the latter is utilized to extract multiple resolutions of features and fuse the features with a feature balance kernel for rich representation. Extensive experiments on several image recognition tasks demonstrate the superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Many typical applications of object detection operate within a prescribed false-positive range. In this situation the performance of a detector should be assessed on the basis of the area under the ROC curve over that range, rather than over the full curve, as the performance outside the prescribed range is irrelevant. This measure is labelled as the partial area under the ROC curve (pAUC). We propose a novel ensemble learning method which achieves a maximal detection rate at a user-defined range of false positive rates by directly optimizing the partial AUC using structured learning. In addition, in order to achieve high object detection performance, we propose a new approach to extracting low-level visual features based on spatial pooling. Incorporating spatial pooling improves the translational invariance and thus the robustness of the detection process. Experimental results on both synthetic and real-world data sets demonstrate the effectiveness of our approach, and we show that it is possible to train state-of-the-art pedestrian detectors using the proposed structured ensemble learning method with spatially pooled features. The result is the current best reported performance on the Caltech-USA pedestrian detection dataset. [ABSTRACT FROM PUBLISHER]

A variety of existing image quality assessment (IQA) metrics share a similar two-stage framework: at the first stage, a quality map is constructed by comparison between local regions of reference and distorted images; at the second stage, the spatial pooling is adopted to obtain overall quality score. In this work, we propose a novel spatial pooling strategy for image quality assessment through statistical analysis of the quality map. Our in-depth analysis indicates that the overall image quality is sensitive to the quality distribution. Based on the analysis, the quality histogram and statistical descriptors extracted from the quality map are used as input to the support vector regression to obtain the final objective quality score. Experimental results on three large public IQA databases have demonstrated that the proposed spatial pooling strategy can greatly improve the quality prediction performance of the original IQA metrics in terms of correlation with human subjective ratings. [ABSTRACT FROM AUTHOR]

In daily life, humans demonstrate an amazing ability to remember images they see on magazines, commercials, TV, web pages, etc. but automatic prediction of intrinsic memorability of images using computer vision and machine learning techniques has only been investigated very recently. Our goal in this article is to explore the role of visual attention and image semantics in understanding image memorability. In particular, we present an attention-driven spatial pooling strategy and show that considering image features from the salient parts of images improves the results of the previous models. We also investigate different semantic properties of images by carrying out an analysis of a diverse set of recently proposed semantic features which encode meta-level object categories, scene attributes, and invoked feelings. We show that these features which are automatically extracted from images provide memorability predictions as nearly accurate as those derived from human annotations. Moreover, our combined model yields results superior to those of state-of-the art fully automatic models. [ABSTRACT FROM AUTHOR]

We propose a fast quality metric for depth maps, called fast depth quality metric (FDQM), which efficiently evaluates the impacts of depth map errors on the qualities of synthesized intermediate views in multiview video plus depth applications. In other words, the proposed FDQM assesses view synthesis distortions in the depth map domain, without performing the actual view synthesis. First, we estimate the distortions at pixel positions, which are specified by reference disparities and distorted disparities, respectively. Then, we integrate those pixel-wise distortions into an FDQM score by employing a spatial pooling scheme, which considers occlusion effects and the characteristics of human visual attention. As a benchmark of depth map quality assessment, we perform a subjective evaluation test for intermediate views, which are synthesized from compressed depth maps at various bitrates. We compare the subjective results with objective metric scores. Experimental results demonstrate that the proposed FDQM yields highly correlated scores to the subjective ones. Moreover, FDQM requires at least 10 times less computations than conventional quality metrics, since it does not perform the actual view synthesis. [ABSTRACT FROM PUBLISHER]

The output of primary visual cortex (V1) is a piecemeal representation of the visual scene and the response of any one cell cannot unambiguously guide sensorimotor behavior. It remains unsolved how subsequent stages of cortical processing combine (‘pool’) these early visual signals into a coherent representation. We (Webb et al., 2007, 2011) have shown that responses of human observers on a pooling task employing broadband, random dot motion can be accurately predicted by decoding the maximum likelihood direction from a population of motion-sensitive neurons. Whereas Amano et al. (2009) found that the vector average velocity of arrays of narrowband, two-dimensional (2-d) plaids predicts perceived global motion. To reconcile these different results, we designed two experiments in which we used 2-d noise textures moving behind spatially distributed apertures and measured the point of subjective equality between pairs of global noise textures. Textures in the standard stimulus moved rigidly in the same direction, whereas their directions in the comparison stimulus were sampled from a set of probability distributions. Human observers judged which noise texture had a more clockwise global direction. In agreement with Amano and colleagues, observers’ perceived global motion coincided with the vector average stimulus direction. To test if directional ambiguities in local motion signals governed perceived global direction, we manipulated the fidelity of the texture motion within each aperture. A proportion of the apertures contained texture that underwent rigid translation and the remainder contained dynamic (temporally uncorrelated) noise to create locally ambiguous motion. Perceived global motion matched the vector average when the majority of apertures contained rigid motion, but with increasing levels of dynamic noise shifted towards the maximum likelihood direction. A class of population decoders utilizing power-law nonlinearities can accommodate this flexible pooling.

Visual sequential search might use a peripheral spatial ranking of the scene to put the next target of the sequence in the correct order. This strategy, indeed, might enhance the discriminative capacity of the human peripheral vision and spare neural resources associated with foveation. However, it is not known how exactly the peripheral vision sustains sequential search and whether the sparing of neural resources has a cost in terms of performance. To elucidate these issues, we compared strategy and performance during an alpha-numeric sequential task where peripheral vision was modulated in three different conditions: normal, blurred, or obscured. If spatial ranking is applied to increase the peripheral discrimination, its use as a strategy in visual sequencing should differ according to the degree of discriminative information that can be obtained from the periphery. Moreover, if this strategy spares neural resources without impairing the performance, its use should be associated with better performance. We found that spatial ranking was applied when peripheral vision was fully available, reducing the number and time of explorative fixations. When the periphery was obscured, explorative fixations were numerous and sparse; when the periphery was blurred, explorative fixations were longer and often located close to the items. Performance was significantly improved by this strategy. Our results demonstrated that spatial ranking is an efficient strategy adopted by the brain in visual sequencing to highlight peripheral detection and discrimination; it reduces the neural cost by avoiding unnecessary foveations, and promotes sequential search by facilitating the onset of a new saccade. [ABSTRACT FROM AUTHOR]

We present a new method for automatic classification of indirect immunoflourescence images of HEp-2 cells into different staining pattern classes. Our method is based on a new texture measure called shape index histograms that captures second-order image structure at multiple scales. Moreover, we introduce a spatial decomposition scheme which is radially symmetric and suitable for cell images. The spatial decomposition is performed using donut-shaped pooling regions of varying sizes when gathering histogram contributions. We evaluate our method using both the ICIP 2013 and the ICPR 2012 competition datasets. Our results show that shape index histograms are superior to other popular texture descriptors for HEp-2 cell classification. Moreover, when comparing to other automated systems for HEp-2 cell classification we show that shape index histograms are very competitive; especially considering the relatively low complexity of the method. [ABSTRACT FROM AUTHOR]

Abstract: The development of objective image quality assessment (IQA) metrics aligned with human perception is of fundamental importance to numerous image-processing applications. Recently, human visual system (HVS)-based engineering algorithms have received widespread attention for their low computational complexity and good performance. In this paper, we propose a new IQA model by incorporating these available engineering principles. A local singular value decomposition (SVD) is first utilised as a structural projection tool to select local image distortion features, and then, both perceptual spatial pooling and neural networks (NN) are employed to combine feature vectors to predict a single perceptual quality score. Extensive experiments and cross-validations conducted with three publicly available IQA databases demonstrate the accuracy, consistency, robustness, and stability of the proposed approach compared to state-of-the-art IQA methods, such as Visual Information Fidelity (VIF), Visual Signal to Noise Ratio (VSNR), and Structural Similarity Index (SSIM). [Copyright &y& Elsevier]

Goal-driven top-down mechanism plays important role in the case of object detection and recognition. In this paper, we propose a top-down computational model for goal-driven saliency detection based on the coding-based classification framework. It consists of four successive steps: feature extraction, descriptor coding, contextual pooling and saliency prediction. Particularly, we investigate the effect of spatial context information for saliency detection, and propose a block-wise spatial pooling operation to take advantage of contextual cues in multiple neighborhood scales and orientations. The experimental results on three datasets demonstrate that our method can effectively exploit contextual information and achieves the state-of-the-art performance on top-down saliency detection task. [ABSTRACT FROM AUTHOR]

The output of primary visual cortex (V1) is a piecemeal representation of the visual scene and the response of any one cell cannot unambiguously guide sensorimotor behavior. It remains unsolved how subsequent stages of cortical processing combine ("pool") these early visual signals into a coherent representation. We (Webb et al., 2007, 2011) have shown that responses of human observers on a pooling task employing broadband, random dot motion can be accurately predicted by decoding the maximum likelihood direction from a population of motion-sensitive neurons. Whereas Amano et al. (2009) found that the vector average velocity of arrays of narrowband, two-dimensional (2-d) plaids predicts perceived global motion. To reconcile these different results, we designed two experiments in which we used 2-d noise textures moving behind spatially distributed apertures and measured the point of subjective equality between pairs of global noise textures. Textures in the standard stimulus moved rigidly in the same direction, whereas their directions in the comparison stimulus were sampled from a set of probability distributions. Human observers judged which noise texture had a more clockwise (CW) global direction. In agreement with Amano and colleagues, observers' perceived global motion coincided with the vector average stimulus direction. To test if directional ambiguities in local motion signals governed perceived global direction, we manipulated the fidelity of the texture motion within each aperture. A proportion of the apertures contained texture that underwent rigid translation and the remainder contained dynamic (temporally uncorrelated) noise to create locally ambiguous motion. Perceived global motion matched the vector average when the majority of apertures contained rigid motion, but with increasing levels of dynamic noise shifted toward the maximum likelihood direction. A class of population decoders utilizing power-law non-linearities can accommodate this flexible pooling. [ABSTRACT FROM AUTHOR]

Abstract: Many objective image quality assessment algorithms firstly apply quality metrics in local regions that results in a quality map, and then pool the quality values in the quality map into a single quality score. The simplest pooling method is the average of quality values, which assumes that all the quality values are independent and equally important. However, visual perception is so complex that the assumption underlying average pooling might be too strict. There is an agreement that some regions in the images might be more perceptually significant, which leads to more advanced spatial pooling methods. In this work we evaluate existing spatial pooling methods for five important quality attributes, which are proposed to reduce the complexity of image quality assessment. The results show that: (1) more advanced spatial pooling methods are generally better than simple average; (2) spatial pooling depends on both image quality metrics and the attributes of the image. [Copyright &y& Elsevier]

• We revisit the squeeze operation in SENets, and shed lights on why and how to embed rich (both global and local) spatial information into the excitation module to improve accuracy. • We propose an integrated two-stage spatial pooling method with two efficient implementation approaches for rich descriptor extraction. • We conduct extensive experiments to verify convincing improvements over SENets and their extension on various fundamental computer vision tasks. Squeeze-and-Excitation (SE) blocks have demonstrated significant accuracy gains for state-of-the-art deep architectures by re-weighting channel-wise feature responses. The SE block is an architecture unit that integrates two operations: a squeeze operation that employs global average pooling to aggregate spatial convolutional features into a channel feature, and an excitation operation that learns instance-specific channel weights from the squeezed feature to re-weight each channel. In this paper, we revisit the squeeze operation in SE blocks, and shed lights on why and how to embed rich (both global and local) information into the excitation module at minimal extra costs. In particular, we introduce a simple but effective two-stage spatial pooling process: rich descriptor extraction and information fusion. The rich descriptor extraction step aims to obtain a set of diverse (i.e., global and especially local) deep descriptors that contain more informative cues than global average-pooling. While, absorbing more information delivered by these descriptors via a fusion step can aid the excitation operation to return more accurate re-weight scores in a data-driven manner. We validate the effectiveness of our method by extensive experiments on ImageNet for image classification and on MS-COCO for object detection and instance segmentation. For these experiments, our method achieves consistent improvements over the SENets on all tasks, in some cases, by a large margin. [ABSTRACT FROM AUTHOR]

Abstract: Saccades aimed at spatially extended targets land reliably at central locations determined by pooling information across the target shape [Melcher, D., & Kowler, E. (1999). Shape, surfaces and saccades. Vision Research, 39, 2929–2946; Vishwanath, D., & Kowler, E. (2003). Localization of shapes: Eye movements and perception compared. Vision Research, 43, 1637–1653]. Previous findings of saccadic errors when attempting to look at a target in the midst of distractors encouraged suggestions that pooling occurs indiscriminately, with little or no influence of a selective filter to eliminate the influence of nearby distractors. To determine the effectiveness of filtering, saccadic localization was studied for saccades made to a set of target elements (discs) interleaved with an equivalent set of distractors of a different color. With such interleaved elements, selection and spatial pooling are constrained to occur over the same spatial region. The results showed that filtering was effective and saccadic landing position was determined mainly by the target elements. Concurrent perceptual judgments made about the same stimuli (estimating the mean size of either target or distractor discs) showed better performance for the target discs than distractors, confirming that perceptual attention was allocated to the set of target elements. These results: (1) support the role of attention in setting the input to the spatial pooling process that guides saccades to spatially extended targets, and (2) show that perceptual judgments of mean value, often thought to impose modest attentional demands, are not immune to the constraints of this pre-saccadic filter. [Copyright &y& Elsevier]

Assuming a convergent projection within a hierarchy of processing stages stimuli from different areas of the receptive field project onto the same population of cells. Pooling over space affects the representation of individual stimuli, and thus its understanding is crucial for attention and ultimately for object recognition. Since attention, in turn, is likely to modify such spatial pooling by changing the competitive weight of individual stimuli, we compare the predictions of sum- and max-pooling methods using a model of attention. Both pooling functions can account for data investigating the competition between a pair of stimuli within a V4 receptive field; however, our model using sum-pooling predicts a different tuning curve. If we present an additional probe stimulus with the pair, sum-pooling predicts a bottom-up bias of attention, whereas the competition for attention using max-pooling is robust against the additional stimulus. [Copyright &y& Elsevier]

• We propose a quaternionic extended local binary pattern for color image representation. • We propose an extended quaternionic representation by introducing an information term into the real part of a quaternion. • We propose adaptive structural pyramid pooling to aggregate encoded features into histogram representations. • Our method achieves the state-of-the-art results for color texture classification and scene categorization. This paper proposes a novel feature representation method for color images, namely quaternionic extended local binary pattern (QxLBP) with adaptive structural pyramid pooling (ASPP). First, we propose a QxLBP operator to encode local neighboring information and complementary modulus and phase information in the quaternion domain of color images. In QxLBP, an extended quaternionic representation (EQR) is proposed which introduces an information term into the real part of a quaternion. The resulting EQR enables us to flexibly encode discriminative features and handle multichannel image data. Second, we propose ASPP as a multiresolution pooling way to aggregate local features. Unlike the traditional spatial pyramid pooling which is sensitive to image rotation and spatial changes, ASPP is structure-oriented pooling which can adaptively aggregate the encoded features into multiresolution histogram representations. Experiments on four benchmark image datasets demonstrate that the proposed method achieves the state-of-the-art performance for color texture classification and scene categorization. [ABSTRACT FROM AUTHOR]

Deep hashing is the mainstream algorithm for large-scale cross-modal retrieval due to its high retrieval speed and low storage capacity, but the problem of reconstruction of modal semantic information is still very challenging. In order to further solve the problem of unsupervised cross-modal retrieval semantic reconstruction, we propose a novel deep semantic-preserving reconstruction hashing (DSPRH). The algorithm combines spatial and channel semantic information, and mines modal semantic information based on adaptive self-encoding and joint semantic reconstruction loss. The main contributions are as follows: (1) We introduce a new spatial pooling network module based on tensor regular-polymorphic decomposition theory to generate rank-1 tensor to capture high-order context semantics, which can assist the backbone network to capture important contextual modal semantic information. (2) Based on optimization perspective, we use global covariance pooling to capture channel semantic information and accelerate network convergence. In feature reconstruction layer, we use two bottlenecks auto-encoding to achieve visual-text modal interaction. (3) In metric learning, we design a new loss function to optimize model parameters, which can preserve the correlation between image modalities and text modalities. The DSPRH algorithm is tested on MIRFlickr-25K and NUS-WIDE. The experimental results show that DSPRH has achieved better performance on retrieval tasks. [ABSTRACT FROM AUTHOR]

We propose a new efficient architecture for semantic segmentation, based on a "Waterfall" Atrous Spatial Pooling architecture, that achieves a considerable accuracy increase while decreasing the number of network parameters and memory footprint. The proposed Waterfall architecture leverages the efficiency of progressive filtering in the cascade architecture while maintaining multiscale fields-of-view comparable to spatial pyramid configurations. Additionally, our method does not rely on a postprocessing stage with Conditional Random Fields, which further reduces complexity and required training time. We demonstrate that the Waterfall approach with a ResNet backbone is a robust and efficient architecture for semantic segmentation obtaining state-of-the-art results with significant reduction in the number of parameters for the Pascal VOC dataset and the Cityscapes dataset. [ABSTRACT FROM AUTHOR]

Under the influence of local‐ and large‐scale climatological processes, extreme river flow events often show long‐term trends, seasonality, interyear variability, and other characteristics of temporal nonstationarity. Properly accounting for this nonstationarity is vital for making accurate predictions of future floods. In this paper, a regional model based on the generalised Pareto distribution is developed for peaks‐over‐threshold river flow data sets when the event sizes are nonstationary. If observations are nonstationary and covariates are available, then extreme‐value (semi)parametric regression models may be used. Unfortunately, the necessary covariates are rarely observed, and if they are, it is often not clear which process, or combination of processes, to include in the model. Within the statistical literature, latent process (or random effects) models are often used in such scenarios. We develop a regional time‐varying random effects model that allows identification of temporal nonstationarity in event sizes by pooling information across all sites in a spatially homogeneous region. The proposed model, which is an instance of a Bayesian hierarchical model, can be used to predict both unconditional extreme events such as the m‐year maximum and extreme events that condition on being in a given year. The estimated random effects may also tell us about likely candidates for the climatological processes that cause nonstationarity in the flood process. The model is applied to UK flood data from 817 stations spread across 81 hydrometric regions. [ABSTRACT FROM AUTHOR]