Search

Instance segmentation has been widely applied in building extraction from remote sensing imagery in recent years, and accurate instance segmentation results are crucial for urban planning, construction and management. However, existing methods for building instance segmentation (BSI) still have room for improvement. To achieve better detection accuracy and superior performance, we introduce a Hybrid Task Cascade (HTC)-based building extraction method, which is more tailored to the characteristics of buildings. As opposed to a cascaded improvement that performs the bounding box and mask branch refinement separately, HTC intertwines them in a joint multilevel process. The experimental results also validate its effectiveness. Our approach achieves better detection accuracy compared to mainstream instance segmentation methods on three different building datasets, yielding outcomes that are more in line with the distinctive characteristics of buildings. Furthermore, we evaluate the effectiveness of each module of the HTC for building extraction and analyze the impact of the detection threshold on the model's detection accuracy. Finally, we investigate the generalization ability of the proposed model. [ABSTRACT FROM AUTHOR]

Cracks in tunnel linings are the most common tunnel defects. As early indicators of structural deterioration, cracks represent critical problems for the safety of tunnels. Several mobile tunnel inspection systems (MTISs) have been developed for tunnel crack inspection. However, due to the weak signals of cracks, these MTISs require considerable exposure time to capture high-quality tunnel images, necessitating a low travel speed. Meanwhile, traditional crack detection methods encounter difficulties in processing tunnel crack images because of their low contrast and poor continuity. To overcome these challenges, this study presents a new MTIS for fast tunnel crack inspection that consists of a novel mobile imaging module and an automatic crack detection module. The imaging module is composed of an array of high-resolution charge-coupled device (CCD) cameras, a mobile laser scanner, and a lighting array. The core of the crack detection module is a novel lightweight convolutional neural network (CNN) designed for efficient tunnel crack detection, with an effective spatial constraint strategy to guarantee crack continuity. We collected a new tunnel crack dataset consisting of 1,218 images using our mobile imaging module at a driving speed of 80 km/h. Comprehensive experiments were conducted on this dataset to evaluate the performance of our proposed network. The results demonstrate that the presented CNN can effectively detect tunnel cracks with state-of-the-art performance, achieving an F1-score greater than 0.88 and an inference speed of 17 FPS with only 3.4M model parameters. The code and data are available at https://github.com/urban-informatics/LinkCrack. [ABSTRACT FROM AUTHOR]

Estimating similarity using multiple similarity measures or machine learning prediction models is a popular solution to the link prediction problem. The Relation Pattern Deep Learning Classification (RPDLC) technique is proposed in this study, and it is based on multiple neighbor-based similarity metrics and convolution neural networks. The RPDLC first calculates the characteristics for a pair of nodes using neighbor-based metrics and impact nodes. Second, the RPDLC creates a heat map using node characteristics to assess the similarity of the nodes' connection patterns. Third, the RPDLC uses convolution neural network architecture to build a prediction model for missing relationship prediction. On three separate social network datasets, this method is compared to other state-of-the-art algorithms. On all three datasets, the suggested method achieves the greatest AUC, hovering around 99 percent. The use of convolution neural networks and features via relational patterns to create a prediction model are the paper's primary contributions. [ABSTRACT FROM AUTHOR]

Pansharpening is a technology involving information integration and processing in remote sensing imagery. It is applied to generate a high-resolution multispectral (HRMS) image through an effective fusion of a low spatial resolution multispectral image and a panchromatic (PAN) image. In this article, we propose an end-to-end multiscale and multidistillation dilated network (MMDN) for pansharpening. In MMDN, to extract more abundant spatial details from source images, a clique structure-based multiscale dilated block (CSMDB) is presented. The clique structure in CSMDB can fully transfer the information between feature maps obtained by the multiscale dilated convolutional filters. Then, a multidistillation residual information block (MRIB) is constructed to help the network capture the spatial structure of different scales in MS and PAN images. Finally, to reuse and supplement the feature information, a feature embedding strategy is designed by feeding the sum result of the output of cascaded CSMDBs and the shallow features to each MRIB. Experimental results verify that the proposed MMDN outperforms other compared state-of-the-art approaches in terms of objective and subjective evaluations. [ABSTRACT FROM AUTHOR]

Dynamic human activity intensity information is of great importance in many location-based applications. However, two limitations remain in the prediction of human activity intensity. First, it is hard to learn the spatial interaction patterns across scales for predicting human activities. Second, social interaction can help model the activity intensity variation but is rarely considered in the existing literature. To mitigate these limitations, we proposed a novel dynamic activity intensity prediction method with deep learning on graphs using the interactions in both physical and social spaces. In this method, the physical interactions and social interactions between spatial units were integrated into a fused graph convolutional network to model multi-type spatial interaction patterns. The future activity intensity variation was predicted by combining the spatial interaction pattern and the temporal pattern of activity intensity series. The method was verified with a country-scale anonymized mobile phone dataset. The results demonstrated that our proposed deep learning method with combining graph convolutional networks and recurrent neural networks outperformed other baseline approaches. This method enables dynamic human activity intensity prediction from a more spatially and socially integrated perspective, which helps improve the performance of modeling human dynamics. [ABSTRACT FROM AUTHOR]

An introduction is presented in which the editor discusses articles in the issue on topics including cutting edge AutoML research; tree-structured covariance function; and comprehensive study on predicting run time performance under different AutoML.

In this paper, we propose a deep CNN to tackle the image restoration problem by learning formatted information. Previous deep learning based methods directly learn the mapping from corrupted images to clean images, and may suffer from the gradient exploding/vanishing problems of deep neural networks. We propose to address the image restoration problem by learning the structured details and recovering the latent clean image together, from the shared information between the corrupted image and the latent image. In addition, instead of learning the pure difference (corruption), we propose to add a residual formatting layer and an adversarial block to format the information to structured one, which allows the network to converge faster and boosts the performance. Furthermore, we propose a cross-level loss net to ensure both pixel-level accuracy and semantic-level visual quality. Evaluations on public datasets show that the proposed method performs favorably against existing approaches quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

Urban region function recognition is key to rational urban planning and management. Due to the complex socioeconomic nature of functional land use, recognizing urban region function in high-density cities using remote sensing images alone is difficult. The inclusion of social sensing has the potential to improve the function classification performance. However, effectively integrating the multi-source and multi-modal remote and social sensing data remains technically challenging. In this paper, we have proposed a novel end-to-end deep learning-based remote and social sensing data fusion model to address this issue. Two neural network based methods, one based on a 1-dimensional convolutional neural network (CNN) and the other based on a long short-term memory (LSTM) network, have been developed to automatically extract discriminative time-dependent social sensing signature features, which are fused with remote sensing image features extracted via a residual neural network. One of the major difficulties in exploiting social and remote sensing data is that the two data sources are asynchronous. We have developed a deep learning-based strategy to address this missing modality problem by enforcing cross-modal feature consistency (CMFC) and cross-modal triplet (CMT) constraints. We train the model in an end-to-end manner by simultaneously optimizing three costs, including the classification cost, the CMFC cost and the CMT cost. Extensive experiments have been conducted on publicly available datasets to demonstrate the effectiveness of the proposed method in fusing remote and social sensing data for urban region function recognition. The results show that the seemingly unrelated physically sensed image data and social activities sensed signatures can indeed complement each other to help enhance the accuracy of urban region function recognition. [ABSTRACT FROM AUTHOR]

For a city to be livable and walkable is the ultimate goal of future cities. However, conflicts among pedestrians, vehicles, and cyclists at traffic intersections are becoming severe in high-density urban transportation areas, especially in China. Correspondingly, the transit time at intersections is becoming prolonged, and pedestrian safety is becoming endangered. Simulating pedestrian movements at complex traffic intersections is necessary to optimize the traffic organization. We propose an unmanned aerial vehicle (UAV)-based method for tracking and simulating pedestrian movements at intersections. Specifically, high-resolution videos acquired by a UAV are used to recognize and position moving targets, including pedestrians, cyclists, and vehicles, using the convolutional neural network. An improved social force-based motion model is proposed, considering the conflicts among pedestrians, cyclists, and vehicles. In addition, maximum likelihood estimation is performed to calibrate an improved social force model. UAV videos of intersections in Shenzhen are analyzed to demonstrate the performance of the presented approach. The results demonstrate that the proposed social force-based motion model can effectively simulate the movement of pedestrians and cyclists at road intersections. The presented approach provides an alternative method to track and simulate pedestrian movements, thus benefitting the organization of pedestrian flow and traffic signals controlling the intersections. [ABSTRACT FROM AUTHOR]

Urban land use is key to rational urban planning and management. Traditional land use classification methods rely heavily on domain experts, which is both expensive and inefficient. In this paper, deep neural network-based approaches are presented to label urban land use at pixel level using high-resolution aerial images and ground-level street view images. We use a deep neural network to extract semantic features from sparsely distributed street view images and interpolate them in the spatial domain to match the spatial resolution of the aerial images, which are then fused together through a deep neural network for classifying land use categories. Our methods are tested on a large publicly available aerial and street view images dataset of New York City, and the results show that using aerial images alone can achieve relatively high classification accuracy, the ground-level street view images contain useful information for urban land use classification, and fusing street image features with aerial images can improve classification accuracy. Moreover, we present experimental studies to show that street view images add more values when the resolutions of the aerial images are lower, and we also present case studies to illustrate how street view images provide useful auxiliary information to aerial images to boost performances. [ABSTRACT FROM AUTHOR]

• Large-scale and high-resolution approach sensing urban soundscapes without ground measurements with machine learning. • A dataset of street view imagery tagged with multivariate soundscape indicators. • Quantifying the relationships between visual features and human soundscape perception. • Validation with sound intensity and corresponding street view imagery using noise meters and cameras. • Comparative analysis including two cities and a scalable approach. A healthy acoustic environment is an essential component of sustainable cities. Various noise monitoring and simulation techniques have been developed to measure and evaluate urban sounds. However, sensing large areas at a fine resolution remains a great challenge. Based on machine learning, we introduce a new application of street view imagery — estimating large-area high-resolution urban soundscapes, investigating the premise that we can predict and characterize soundscapes without laborious and expensive noise measurements. First, visual features are extracted from street-level imagery using computer vision. Second, fifteen soundscape indicators are identified and a survey is conducted to gauge them solely from images. Finally, a prediction model is constructed to infer the urban soundscape by modeling the non-linear relationship between them. The results are verified with extensive field surveys. Experiments conducted in Singapore and Shenzhen using half a million images affirm that street view imagery enables us to sense large-scale urban soundscapes with low cost but high accuracy and detail, and provides an alternative means to generate soundscape maps. R 2 reaches 0.48 by evaluating the predicted results with field data collection. Further novelties in this domain are revealing the contributing visual elements and spatial laws of soundscapes, underscoring the usability of crowdsourced data, and exposing international patterns in perception. [ABSTRACT FROM AUTHOR]

Accurate and robust short-term bus travel prediction facilitates operating the bus fleet to provide comfortable and flexible bus services. The built environment, including land use, buildings, and public facilities, has an important influence on bus travel demand prediction. However, previous studies regarded the built environment as a static feature thus even ignored its influence on bus travel in deep learning framework. To fill this gap, we propose a graph deep learning-based approach coupling with spatiotemporal influence of built environment (GDLBE) to enhance short-term bus travel demand prediction. A time-dependent geographically weighted regression method is used to resolve the dynamic influence of the built environment on bus travel demand at different times of the day. A graph deep learning module is used to capture the comprehensive spatial and temporal dependency behind massive bus travel demand. The short-term bus travel demand is predicted by fusing the dynamic built environment influences and spatiotemporal dependency. An experiment in Shenzhen is conducted to evaluate the performance of the proposed approach. Baseline methods are compared, and the results demonstrate that the proposed approach outperforms the baselines. These results will help bus fleet dispatch for smart transportation. • Coupling graph deep learning and dynamic built environment influences to forecast short-term bus travel demand. • Multitime GWR is used to reveal that dynamic influence of built environment on bus travel demand. • Deep graph neural network is designed to capture spatial and temporal dependency. • Experiment in Shenzhen demonstrates that the GDLBE approach outperform baseline methods. [ABSTRACT FROM AUTHOR]

Street-level imagery has emerged as a valuable tool for observing large-scale urban spaces with unprecedented detail. However, previous studies have been limited to analyzing individual street-level images. This approach falls short in representing the characteristics of a spatial unit, such as a street or grid, which may contain varying numbers of street-level images ranging from several to hundreds. As a result, a more comprehensive and representative approach is required to capture the complexity and diversity of urban environments at different spatial scales. To address this issue, this study proposes a deep learning-based module called Vision-LSTM, which can effectively obtain vector representation from varying numbers of street-level images in spatial units. The effectiveness of the module is validated through experiments to recognize urban villages, achieving reliable recognition results (overall accuracy: 91.6%) through multimodal learning that combines street-level imagery with remote sensing imagery and social sensing data. Compared to existing image fusion methods, Vision-LSTM demonstrates significant effectiveness in capturing associations between street-level images. The proposed module can provide a more comprehensive understanding of urban spaces, enhancing the research value of street-level imagery and facilitating multimodal learning-based urban research. Our models are available at https://github.com/yingjinghuang/Vision-LSTM. • Representing regional features by capturing associations among street-level images. • The proposed Vision-LSTM extract features from varying numbers of images. • Multimodal model fused satellite imagery, street-level imagery, and mobility data. • Both visual and dynamic mobility information crucial for urban village recognition. • The framework achieved 91.6% accuracy in identifying urban villages. [ABSTRACT FROM AUTHOR]