Search

Vehicle detection in Unmanned Aerial Vehicle (UAV) imagery plays a crucial role in a variety of applications. However, UAVs are usually small, very maneuverable, and can take images from a variety of viewpoints and heights, leading to large differences in vehicle appearance and size. To address the vehicle detection challenge with such diversity in UAV images, we seek to align features between different viewpoints, illumination, weather, and background using remote sensing imagery as an anchor. Following this domain adaptation concept, we propose a multi-scale adversarial network, consisting of a deep convolutional feature extractor, a multi-scale discriminator, and a vehicle detection network. Specifically, the feature extractor is a Siamese network with one path for the UAV imagery and another for the satellite imagery. The shared weights in this sub-network allow us to exploit the large collections of labeled remote sensing imagery for improved vehicle detection in UAV imagery. Experimental results suggest that our proposed algorithm improves the vehicle detection accuracy in the UAVDT dataset and VisDrone dataset. The proposed model achieves great performance in images taken from different perspectives, at different altitudes, and under different imaging situations.

This report describes the development and finalization of the 28th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL 2020), held virtually online, November 3-6, 2020. The attendance for the 2020 conference was 999, the highest in the history of ACM SIGSPATIAL.

This paper addresses the challenge of learning spatial context for the semantic segmentation of high-resolution aerial images using Deep Convolutional Neural Networks (DCNNs). The proposed solution involves deriving a signed distance map for each semantic class from a ground truth label map and training a DCNN to predict this distance map instead of a score map for each class. Since the distance between a target pixel and its nearest object boundary measures how far the pixel penetrates an object, the distance maps encode spatial context, particularly spatial smoothness. Positive pixel values in the distance maps correspond to the correct class and negative values correspond to the incorrect class. A final label map is derived from the predicted distance maps by selecting the class with the maximum distance. Since neighboring pixels in the distance maps have similar values, the segmentation results are smoother than current approaches. The results are shown to be even better than performing post-processing using fully connected Conditional Random Fields (CRFs), a common approach to smoothing the segmentations produced DCNNs. Experimental results on the semantic labeling challenge dataset show the proposed approach outperforms most state-of-the-art methods. Our main contribution, though, is the novel idea of replacing the pixel-wise class score maps of DCNNs with distance maps. This is therefore orthogonal and complementary to other techniques employed by the state-of-the-art methods and could therefore be used to improve upon them.

Geospatial artificial intelligence (GeoAI) is an interdisciplinary field that has received tremendous attention from both academia and industry in recent years. This article reviews the series of GeoAI workshops held at the Association for Computing Machinery (ACM) International Conference on Advances in Geographic Information Systems (SIGSPATIAL) since 2017. These workshops have provided researchers a forum to present GeoAI advances covering a wide range of topics, such as geospatial image processing, transportation modeling, public health, and digital humanities. We provide a summary of these topics and the research articles presented at the 2017, 2018, and 2019 GeoAI workshops. We conclude with a list of open research directions for this rapidly advancing field.

We propose a robust framework for estimating the ground sample distance (GSD) of overhead images using off-the-shelf object detectors. Knowing the GSD of an overhead image is important for many tasks yet images are increasingly being distributed without such meta data due to the advent of drones. We make our framework robust by performing detections over a number of scales and orientations. We derive a reference curve from a large number of training images that relates GSD to object size. This reference curve is used to estimate the GSD of a test image. We demonstrate the effectiveness of our framework on drone imagery ranging from 2 to 32cm GSD using an off-the-shelf car detector.

This report describes the development and finalization of the 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL 2019), held in Chicago, Illinois, USA, November 5-8, 2019. The attendance for the 2019 conference was 397, the second highest in the history of ACM SIGSPATIAL. Historically, what is now the ACM SIGSPATIAL conference started as a series of workshops and symposia in 1993. Its aim was to promote the interdisciplinary discussions among researchers, developers, users, and practitioners and fostering research in all aspects of Geographic Information Systems --- hence the original workshop acronym ACM GIS. The focus was on novel systems based on geospatial data and knowledge. It continued its mission of providing a forum for original research results, addressing conceptual, design, and implementation aspects of geospatial data ranging from applications, user interfaces and visualization, to data storage, query processing, indexing and data mining. The conference is now the premier annual event of the ACM Special Interest Group on Spatial Information (ACM SIGSPATIAL).

Multimedia researchers have exploited large collections of community-contributed geo-referenced images to better understand a particular image, such as its subject matter or where it was taken, as well as to better understand a geographic location, such as the most visited tourist spots in a city or what the local cuisine is like. The goal of this paper is to better understand location. In particular, we use geo-referenced image collections to better understand what occurs in different parts of a city at fine spatial and activity class scales. This problem is known as land use mapping in the geographical sciences. We propose a novel framework to perform fine-grained land use mapping at the city scale using ground-level images. Mapping land use is considerably more difficult than mapping land cover and is generally not possible using overhead imagery as it requires close-up views and seeing inside buildings. We postulate that the growing collections of geo-referenced, ground-level images suggest an alternate approach to this geographic knowledge discovery problem. We develop a general framework that uses Flickr images to map 45 different land-use classes for the city of San Francisco, CA, USA. Individual images are classified using a novel convolutional neural network containing two streams: one for recognizing objects and another for recognizing scenes. This network is trained in an end-to-end manner directly on the labeled training images. We propose several novel strategies to overcome the noisiness of our user-generated data including search-based training set augmentation and online adaptive training. We derive a ground truth map of San Francisco in order to evaluate our method. We demonstrate the effectiveness of our approach through geovisualization and quantitative analysis. Our framework achieves over 29% recall at the individual land parcel level that represents a strong baseline for the challenging 45-way land use classification problem, especially given the noisiness of the image data.

This paper formulates cloud and cloud shadow detection as a semantic segmentation problem and proposes a deep convolutional neural network (CNN) based method to detect them in Landsat imagery. Different from traditional machine learning methods, deep CNN-based methods convolve the entire input image to extract multi-level spatial and spectral features, and then deconvolve these features to produce the detailed segmentation. In this way, multi-level features from the whole image and all the bands are utilized to label each pixel as cloud, thin cloud, cloud shadow or clear. An adaption of SegNet with 13 convolutional layers and 13 deconvolution layers is proposed in this study. The method is applied to 38 Landsat 7 images and 32 Landsat 8 images which are globally distributed and have pixel-wise cloud and cloud shadow reference masks provided by the U.S. Geological Survey (USGS). In order to process such large images using the adapted SegNet model on a desktop computer, the Landsat Collection 1 scenes are split into non-overlapping 512 * 512 30 m pixel image blocks. 60% of these blocks are used to train the model using the backpropagation algorithm, 10% of the blocks are used to validate the model and tune its parameters, and the remaining 30% of the blocks are used for performance evaluation. Compared with the cloud and cloud shadow masks produced by CFMask, which are provided with the Landsat Collection 1 data, the overall accuracies are significantly improved from 89.88% and 84.58% to 95.26% and 95.47% for the Landsat 7 and Landsat 8 images respectively. The proposed method benefits from the multi-level spatial and spectral features, and results in more than a 40% increase in user's accuracy and in more than a 20% increase in producer's accuracy for cloud shadow detection in Landsat 8 imagery. The issues for operational implementation are discussed.

Remote sensing image retrieval(RSIR), which aims to efficiently retrieve data of interest from large collections of remote sensing data, is a fundamental task in remote sensing. Over the past several decades, there has been significant effort to extract powerful feature representations for this task since the retrieval performance depends on the representative strength of the features. Benchmark datasets are also critical for developing, evaluating, and comparing RSIR approaches. Current benchmark datasets are deficient in that 1) they were originally collected for land use/land cover classification and not image retrieval, 2) they are relatively small in terms of the number of classes as well the number of sample images per class, and 3) the retrieval performance has saturated. These limitations have severely restricted the development of novel feature representations for RSIR, particularly the recent deep-learning based features which require large amounts of training data. We therefore present in this paper, a new large-scale remote sensing dataset termed "PatternNet" that was collected specifically for RSIR. PatternNet was collected from high-resolution imagery and contains 38 classes with 800 images per class. We also provide a thorough review of RSIR approaches ranging from traditional handcrafted feature based methods to recent deep learning based ones. We evaluate over 35 methods to establish extensive baseline results for future RSIR research using the PatternNet benchmark., Comment: 49 pages

Land-cover classification using remote sensing imagery is an important Earth observation task. Recently, land cover classification has benefited from the development of fully connected neural networks for semantic segmentation. The benchmark datasets available for training deep segmentation models in remote sensing imagery tend to be small, however, often consisting of only a handful of images from a single location with a single scale. This limits the models' ability to generalize to other datasets. Domain adaptation has been proposed to improve the models' generalization but we find these approaches are not effective for dealing with the scale variation commonly found between remote sensing image collections. We therefore propose a scale aware adversarial learning framework to perform joint cross-location and cross-scale land-cover classification. The framework has a dual discriminator architecture with a standard feature discriminator as well as a novel scale discriminator. We also introduce a scale attention module which produces scale-enhanced features. Experimental results show that the proposed framework outperforms state-of-the-art domain adaptation methods by a large margin., The open-sourced codes are available on Github: https://github.com/xdeng7/scale-aware_da

Overhead image geolocalization is becoming increasingly important due to the growing collection of drone imagery without location information. In this paper, we perform large-scale overhead image geolocalization by matching a query image to wide-area reference imagery with known location. We use deep local features so that the query image need not align with but only overlap the tiled reference imagery. We further address two key challenges. For when the query and reference imagery are from different dates, we perform cross-time geolocalization using time invariant features learned using a Siamese network. For when the query and reference imagery are oriented differently, we introduce an orientation normalization network. We demonstrate our contributions on two new high-resolution overhead image datasets. Our method significantly outperforms strong baselines on cross-time geolocalization and is shown to exhibit promising orientation invariance.

We investigate the problem of estimating the spatial resolution of overhead imagery. More overhead imagery is becoming available without such meta-data either because it was not collected in the first place or was not preserved with the imagery. Knowing the spatial resolution can be important for a range of automated image understanding tasks such as object detection, semantic segmentation, etc. In this paper, we explore a regression framework with a feature extraction frontend and a dilated convolution backend to estimate the spatial resolution of an overhead image. We show that a stacked auto-encoder frontend outperforms a standard convolution neural network feature extractor. In order to demonstrate our approach, we construct an evaluation dataset consisting of a large collection of very high-resolution overhead images with spatial resolutions ranging from 0.15 to 1.0 meters per pixel.

Many applications of remote sensing only require the classification of a single land type. This is known as the one-class classification problem and it can be performed using either binary classifi...

Nowadays artificial intelligence (AI) is bringing tremendous opportunities and challenges to geospatial research. Big data enable computers to observe and learn the world from many different perspectives, while high performance machines support the development, training, and deployment of AI models within reasonable amount of time. Recent years have witnessed significant advances in the integration of geospatial study and AI in both academia and industry. There have already been many successful studies for both physical environment and human society. Focusing on modeling the physical nature, research has shown that deep learning can improve the representation of clouds that are smaller than the grid resolutions of climate models. Examining the human society, AI and natural language processing methods, such as word embeddings, help quantify changes in stereotypes and attitudes toward women and ethnic minorities over 100 years in the United States. There are also many other applications that effectively integrate AI with problems in geospatial studies, such as vehicle trajectory prediction, high-definition mapping and navigation, historical map digitizing, gazetteer conflation, geographic feature extraction, and place understanding. The 3nd International Workshop on AI for Geographic Knowledge Discovery (GeoAI 2019) builds on the success of the previous workshops in 2017 and 2018. GeoAI is bringing together geoscientists, computer scientists, engineers, entrepreneurs, and decision makers from academia, industry, and government to discuss the latest trends, successes, challenges, and opportunities in the field of artificial intelligence for data mining and geographic knowledge discovery.

We focus on the novel problem of estimating the spatial resolution of overhead imagery. More and more overhead imagery is becoming available without such meta-data either because it was not collected in the first place or was not preserved with the imagery. We propose a bottom-up, data-driven approach using convolutional neural networks. We show that an extended model which incorporates dilated convolution to expand the receptive field of the network outperforms a baseline model on an evaluation dataset with a range of simulated spatial resolutions. We make a number of interesting observations to motivate future work on this novel problem.

Semantic segmentation requires large amounts of pixel-wise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks. We exploit video prediction models' ability to predict future frames in order to also predict future labels. A joint propagation strategy is also proposed to alleviate mis-alignments in synthesized samples. We demonstrate that training segmentation models on datasets augmented by the synthesized samples leads to significant improvements in accuracy. Furthermore, we introduce a novel boundary label relaxation technique that makes training robust to annotation noise and propagation artifacts along object boundaries. Our proposed methods achieve state-of-the-art mIoUs of 83.5% on Cityscapes and 82.9% on CamVid. Our single model, without model ensembles, achieves 72.8% mIoU on the KITTI semantic segmentation test set, which surpasses the winning entry of the ROB challenge 2018. Our code and videos can be found at https://nv-adlr.github.io/publication/2018-Segmentation., CVPR 2019 Oral. Code link: https://github.com/NVIDIA/semantic-segmentation. YouTube link: https://www.youtube.com/watch?v=aEbXjGZDZSQ

Current state-of-the-art approaches to video understanding adopt temporal jittering to simulate analyzing the video at varying frame rates. However, this does not work well for multirate videos, in which actions or subactions occur at different speeds. The frame sampling rate should vary in accordance with the different motion speeds. In this work, we propose a simple yet effective strategy, termed random temporal skipping, to address this situation. This strategy effectively handles multirate videos by randomizing the sampling rate during training. It is an exhaustive approach, which can potentially cover all motion speed variations. Furthermore, due to the large temporal skipping, our network can see video clips that originally cover over 100 frames. Such a time range is enough to analyze most actions/events. We also introduce an occlusion-aware optical flow learning method that generates improved motion maps for human action recognition. Our framework is end-to-end trainable, runs in real-time, and achieves state-of-the-art performance on six widely adopted video benchmarks.

Deep neural networks have led to a series of breakthroughs in computer vision given sufficient annotated training datasets. For novel tasks with limited labeled data, the prevalent approach is to transfer the knowledge learned in the pre-trained models to the new tasks by fine-tuning. Classic model fine-tuning utilizes the fact that well trained neural networks appear to learn cross domain features. These features are treated equally during transfer learning. In this paper, we explore the impact of feature selection in model fine-tuning by introducing a transfer module, which assigns weights to features extracted from pretrained models. The proposed transfer module proves the importance of feature selection for transferring models from source to target domains. It is shown to significantly improve upon fine-tuning results with only marginal extra computational cost. We also incorporate an auxiliary classifier as an extra regularizer to avoid over-fitting. Finally, we build a Gated Transfer Network (GTN) based on our transfer module and achieve state-of-the-art results on six different tasks.

Analyzing videos of human actions involves understanding the temporal relationships among video frames. State-of-the-art action recognition approaches rely on traditional optical flow estimation methods to pre-compute motion information for CNNs. Such a two-stage approach is computationally expensive, storage demanding, and not end-to-end trainable. In this paper, we present a novel CNN architecture that implicitly captures motion information between adjacent frames. We name our approach hidden two-stream CNNs because it only takes raw video frames as input and directly predicts action classes without explicitly computing optical flow. Our end-to-end approach is 10x faster than its two-stage baseline. Experimental results on four challenging action recognition datasets: UCF101, HMDB51, THUMOS14 and ActivityNet v1.2 show that our approach significantly outperforms the previous best real-time approaches.

This paper investigates conditional generative adversarial networks (cGANs) to overcome a fundamental limitation of using geotagged media for geographic discovery, namely its sparse and uneven spatial distribution. We train a cGAN to generate ground-level views of a location given overhead imagery. We show the "fake" ground-level images are natural looking and are structurally similar to the real images. More significantly, we show the generated images are representative of the locations and that the representations learned by the cGANs are informative. In particular, we show that dense feature maps generated using our framework are more effective for land-cover classification than approaches which spatially interpolate features extracted from sparse ground-level images. To our knowledge, ours is the first work to use cGANs to generate ground-level views given overhead imagery and to explore the benefits of the learned representations., Comment: 10 pages, 5 figures, camera-ready version of ACM SIGSPATIAL 2018 (ORAL)

Molecular dynamics (MD) simulation is a powerful technique for sampling the conformational landscape of natively folded proteins (NFPs) and structurally dynamic intrinsically disordered proteins (IDPs). NFPs and IDPs can be viewed as nonlinear dynamical systems that exercise available degrees of freedom to explore their energetically-accessible conformation landscape. Dimensionality estimators have emerged as useful tools to characterize nonlinear dynamical systems in other domains, but their application to MD simulation has been limited due to thermal noise and a lack of ground-truth data. We develop a series of increasingly complex biopolymer models which exhibit a range of dynamics we seek to characterize in MD simulations (stochastic dynamics, helical structures, partially folded states, and correlated motions) and are of known dimensionality. We utilize the maximum-likelihood dimension (MLD) estimator to investigate the effects of thermal noise and noise-smoothing techniques on the estimates obtained from the polymer models and MD simulations of two NFPs and two IDPs. We find that under certain noise/smoothing conditions, the MLD over/under-estimates the true dimensionality of the models in a predictable manner, allowing us to relate differences between MLD estimates to differences between NFP and IDP motions for classification of biomolecular systems based on their dynamics.

We argue that georeferenced social multimedia is really a form of volunteered geographic information. For example, community-contributed images and videos available at websites such as Flickr often indicate the location where they were acquired, and, thus, potentially contain a wealth of information about what-is-where on the surface of the Earth. The challenge is how to extract this information from these complex and noisy data, preferably in an automated fashion. We describe a novel analysis framework termed proximate sensing that makes progress towards this goal by using the visual content of georeferenced ground-level images and videos to extract and map geographically relevant information. We describe several geographic knowledge discovery contexts along with case studies where this new analysis paradigm has the potential to map phenomena not easily observable through other means, if at all.

Despite the significant progress that has been made on estimating optical flow recently, most estimation methods, including classical and deep learning approaches, still have difficulty with multi-scale estimation, real-time computation, and/or occlusion reasoning. In this paper, we introduce dilated convolution and occlusion reasoning into unsupervised optical flow estimation to address these issues. The dilated convolution allows our network to avoid upsampling via deconvolution and the resulting gridding artifacts. Dilated convolution also results in a smaller memory footprint which speeds up interference. The occlusion reasoning prevents our network from learning incorrect deformations due to occluded image regions during training. Our proposed method outperforms state-of-the-art unsupervised approaches on the KITTI benchmark. We also demonstrate its generalization capability by applying it to action recognition in video., Accepted at ICIP 2018

In recent years, geotagged social media has become popular as a novel source for geographic knowledge discovery. Ground-level images and videos provide a different perspective than overhead imagery and can be applied to a range of applications such as land use mapping, activity detection, pollution mapping, etc. The sparse and uneven distribution of this data presents a problem, however, for generating dense maps. We therefore investigate the problem of spatially interpolating the high-dimensional features extracted from sparse social media to enable dense labeling using standard classifiers. Further, we show how prior knowledge about region boundaries can be used to improve the interpolation through spatial morphing kernel regression. We show that an interpolate-then-classify framework can produce dense maps from sparse observations but that care must be taken in choosing the interpolation method. We also show that the spatial morphing kernel improves the results., accepted by ICIP 2018

Unseen Action Recognition (UAR) aims to recognise novel action categories without training examples. While previous methods focus on inner-dataset seen/unseen splits, this paper proposes a pipeline using a large-scale training source to achieve a Universal Representation (UR) that can generalise to a more realistic Cross-Dataset UAR (CD-UAR) scenario. We first address UAR as a Generalised Multiple-Instance Learning (GMIL) problem and discover 'building-blocks' from the large-scale ActivityNet dataset using distribution kernels. Essential visual and semantic components are preserved in a shared space to achieve the UR that can efficiently generalise to new datasets. Predicted UR exemplars can be improved by a simple semantic adaptation, and then an unseen action can be directly recognised using UR during the test. Without further training, extensive experiments manifest significant improvements over the UCF101 and HMDB51 benchmarks., Comment: Accepted at CVPR 2018

This paper performs a quantitative comparison of open-source data available on the Internet for the fine-grain mapping of land use. Three points of interest (POI) data sources--Google Places, Bing Maps, and the Yellow Pages--and one volunteered geographic information data source--Open Street Map (OSM)--are compared with each other at the parcel level for San Francisco with respect to a proposed fine-grain land-use taxonomy. The sources are also compared to coarse-grain authoritative data which we consider to be the ground truth. Results show limited agreement among the data sources as well as limited accuracy with respect to the authoritative data even at coarse class granularity. We conclude that POI and OSM data do not appear to be sufficient alone for fine-grain land-use mapping., ACM SIGSPATIAL 2017 Workshop on Urban GIS

We investigate the problem of representing an entire video using CNN features for human action recognition. End-to-end learning of CNN/RNNs is currently not possible for whole videos due to GPU memory limitations and so a common practice is to use sampled frames as inputs along with the video labels as supervision. However, the global video labels might not be suitable for all of the temporally local samples as the videos often contain content besides the action of interest. We therefore propose to instead treat the deep networks trained on local inputs as local feature extractors. The local features are then aggregated to form global features which are used to assign video-level labels through a second classification stage. We investigate a number of design choices for this local feature approach. Experimental results on the HMDB51 and UCF101 datasets show that a simple maximum pooling on the sparsely sampled local features leads to significant performance improvement.

This paper is the first work to perform spatio-temporal mapping of human activity using the visual content of geo-tagged videos. We utilize a recent deep-learning based video analysis framework, termed hidden two-stream networks, to recognize a range of activities in YouTube videos. This framework is efficient and can run in real time or faster which is important for recognizing events as they occur in streaming video or for reducing latency in analyzing already captured video. This is, in turn, important for using video in smart-city applications. We perform a series of experiments to show our approach is able to accurately map activities both spatially and temporally. We also demonstrate the advantages of using the visual content over the tags/titles., Comment: Accepted at ACM SIGSPATIAL 2017

We investigate publicly available geo-referenced photo collections for land cover classification. Mapping land cover is a fundamental task in the geographic sciences and is typically done using remote sensing (overhead) imagery through manual annotation. We here propose a novel alternate approach based on proximate sensing. The goal of proximate sensing is to map what-is-where on the surface of the Earth using ground level images of objects and scenes. It has the potential to map phenomena not observable through remote sensing. We perform an extensive case study on using ground level images for binary land cover classification into developed and undeveloped regions. We investigate visual features and text annotations to label images or sets of images with these two classes. Knowing the location of the images allows us to generate land cover maps which we quantitatively evaluate using ground truth maps. We apply our approach to two photo collections, Flickr, the popular photo sharing website, and the Geograph project, whose goal is to collect geographically informative photos. Comparing these two collections allows us to measure the impact of photographer intent. We utilize a weakly supervised learning framework which eliminates the need for manually labeled training data. We also investigate methods for filtering images that are unlikely to be geographically informative. Our results are promising and validate proximate sensing as a novel alternate approach to geographic discovery.

This paper studies the joint learning of action recognition and temporal localization in long, untrimmed videos. We employ a multi-task learning framework that performs the three highly related steps of action proposal, action recognition, and action localization refinement in parallel instead of the standard sequential pipeline that performs the steps in order. We develop a novel temporal actionness regression module that estimates what proportion of a clip contains action. We use it for temporal localization but it could have other applications like video retrieval, surveillance, summarization, etc. We also introduce random shear augmentation during training to simulate viewpoint change. We evaluate our framework on three popular video benchmarks. Results demonstrate that our joint model is efficient in terms of storage and computation in that we do not need to compute and cache dense trajectory features, and that it is several times faster than its sequential ConvNets counterpart. Yet, despite being more efficient, it outperforms state-of-the-art methods with respect to accuracy., WACV 2017 camera ready, minor updates about test time efficiency

In today's era of big data, advanced algorithms, and immense computational power, artificial intelligence (AI) is bringing tremendous opportunities and challenges to geospatial research. Big data enable computers to observe and learn the world from many different perspectives, while high performance machines support the developing, training, and applying of AI models within reasonable amount of time. Recent years have witnessed significant advances in the integration of geography and AI in both academia and industry. There have already been many successful studies. Focusing on modeling the physical nature, research has shown that deep learning can improve the representation of clouds that are smaller than the grid resolutions of climate models. Examining the human society, AI and natural language processing methods, such as word embeddings, are helping quantify changes in stereotypes and attitudes toward women and ethnic minorities over 100 years in the United States. There are also many other applications that effectively integrate AI with problems in geospatial studies, such as vehicle trajectory prediction, indoor navigation, historical map digitizing, gazetteer conflation, geographic feature extraction, geo-ontologies, and place understanding. The 2nd International Workshop on AI for Geographic Knowledge Discovery (GeoAI 2018) builds on the success of the previous workshop in 2017. GeoAI is bringing together geoscientists, computer scientists, engineers, entrepreneurs, and decision makers from academia, industry, and government to discuss the latest trends, successes, challenges, and opportunities in the field of artificial intelligence for data mining and geographic knowledge discovery.

This paper investigates local invariant features for geographic (overhead) image retrieval. Local features are particularly well suited for the newer generations of aerial and satellite imagery whose increased spatial resolution, often just tens of centimeters per pixel, allows a greater range of objects and spatial patterns to be recognized than ever before. Local invariant features have been successfully applied to a broad range of computer vision problems and, as such, are receiving increased attention from the remote sensing community particularly for challenging tasks such as detection and classification. We perform an extensive evaluation of local invariant features for image retrieval of land-use/land-cover (LULC) classes in high-resolution aerial imagery. We report on the effects of a number of design parameters on a bag-of-visual-words (BOVW) representation including saliency- versus grid-based local feature extraction, the size of the visual codebook, the clustering algorithm used to create the codebook, and the dissimilarity measure used to compare the BOVW representations. We also perform comparisons with standard features such as color and texture. The performance is quantitatively evaluated using a first-of-its-kind LULC ground truth data set which will be made publicly available to other researchers. In addition to reporting on the effects of the core design parameters, we also describe interesting findings such as the performance-efficiency tradeoffs that are possible through the appropriate pairings of different-sized codebooks and dissimilarity measures. While the focus is on image retrieval, we expect our insights to be informative for other applications such as detection and classification.

Land use mapping is a fundamental yet challenging task in geographic science. In contrast to land cover mapping, it is generally not possible using overhead imagery. The recent, explosive growth of online geo-referenced photo collections suggests an alternate approach to geographic knowledge discovery. In this work, we present a general framework that uses ground-level images from Flickr for land use mapping. Our approach benefits from several novel aspects. First, we address the nosiness of the online photo collections, such as imprecise geolocation and uneven spatial distribution, by performing location and indoor/outdoor filtering, and semi- supervised dataset augmentation. Our indoor/outdoor classifier achieves state-of-the-art performance on several bench- mark datasets and approaches human-level accuracy. Second, we utilize high-level semantic image features extracted using deep learning, specifically convolutional neural net- works, which allow us to achieve upwards of 76% accuracy on a challenging eight class land use mapping problem., ACM SIGSPATIAL 2015, Best Poster Award

We perform spatio-temporal analysis of public sentiment using geotagged photo collections. We develop a deep learning-based classifier that predicts the emotion conveyed by an image. This allows us to associate sentiment with place. We perform spatial hotspot detection and show that different emotions have distinct spatial distributions that match expectations. We also perform temporal analysis using the capture time of the photos. Our spatio-temporal hotspot detection correctly identifies emerging concentrations of specific emotions and year-by-year analyses of select locations show there are strong temporal correlations between the predicted emotions and known events., To appear in ACM SIGSPATIAL 2016

This paper performs the first investigation into depth for large-scale human action recognition in video where the depth cues are estimated from the videos themselves. We develop a new framework called depth2action and experiment thoroughly into how best to incorporate the depth information. We introduce spatio-temporal depth normalization (STDN) to enforce temporal consistency in our estimated depth sequences. We also propose modified depth motion maps (MDMM) to capture the subtle temporal changes in depth. These two components significantly improve the action recognition performance. We evaluate our depth2action framework on three large-scale action recognition video benchmarks. Our model achieves state-of-the-art performance when combined with appearance and motion information thus demonstrating that depth2action is indeed complementary to existing approaches.

Learning powerful feature representations for image retrieval has always been a challenging task in the field of remote sensing. Traditional methods focus on extracting low-level hand-crafted features which are not only time-consuming but also tend to achieve unsatisfactory performance due to the complexity of remote sensing images. In this paper, we investigate how to extract deep feature representations based on convolutional neural networks (CNNs) for high-resolution remote sensing image retrieval (HRRSIR). To this end, several effective schemes are proposed to generate powerful feature representations for HRRSIR. In the first scheme, a CNN pre-trained on a different problem is treated as a feature extractor since there are no sufficiently-sized remote sensing datasets to train a CNN from scratch. In the second scheme, we investigate learning features that are specific to our problem by first fine-tuning the pre-trained CNN on a remote sensing dataset and then proposing a novel CNN architecture based on convolutional layers and a three-layer perceptron. The novel CNN has fewer parameters than the pre-trained and fine-tuned CNNs and can learn low dimensional features from limited labelled images. The schemes are evaluated on several challenging, publicly available datasets. The results indicate that the proposed schemes, particularly the novel CNN, achieve state-of-the-art performance.

Nuclear pore complexes (NPCs) gate the only conduits for nucleocytoplasmic transport in eukaryotes. Their gate is formed by nucleoporins containing large intrinsically disordered domains with multiple phenylalanine-glycine repeats (FG domains). In combination, these are hypothesized to form a structurally and chemically homogeneous network of random coils at the NPC center, which sorts macromolecules by size and hydrophobicity. Instead, we found that FG domains are structurally and chemically heterogeneous. They adopt distinct categories of intrinsically disordered structures in non-random distributions. Some adopt globular, collapsed coil configurations and are characterized by a low charge content. Others are highly charged and adopt more dynamic, extended coil conformations. Interestingly, several FG nucleoporins feature both types of structures in a bimodal distribution along their polypeptide chain. This distribution functionally correlates with the attractive or repulsive character of their interactions with collapsed coil FG domains displaying cohesion toward one another and extended coil FG domains displaying repulsion. Topologically, these bipartite FG domains may resemble sticky molten globules connected to the tip of relaxed or extended coils. Within the NPC, the crowding of FG nucleoporins and the segregation of their disordered structures based on their topology, dimensions, and cohesive character could force the FG domains to form a tubular gate structure or transporter at the NPC center featuring two separate zones of traffic with distinct physicochemical properties.

Leveraging large collections of georeferenced, community-contributed photographs can help solve three knowledge-discovery problems: annotating novel images, annotating geographic locations, and performing geographic discovery.

Object based representations of image data enable new contentrelated functionalities while facilitating management of large image databases. Developing such representations for multidate and multi-spectral images is one of the objectives of the second phase of the Alexandria Digital Library (ADL) project at UCSB. Image segmentation and image registration are two of the main issues that are to be addressed in creating localized image representations. We present in this paper some of the recent and current work by the ADL’s image processing group on robust image segmentation, registration, and the use of image texture for content representation. Built upon these technologies are techniques for managing large repositories of data. A t exture thesaurus assists in creating a semantic classification of image regions. An object-based representation is proposed to facilitate data storage, retrieval, analysis, and navigation.

Image texture is useful in image browsing, search and retrieval. A texture descriptor based on a multiresolution decomposition using Gabor wavelets is proposed. The descriptor consists of two parts: a perceptual browsing component (PBC) and a similarity retrieval component (SRC). The extraction methods of both PBC and SRC are based on a multiresolution decomposition using Gabor wavelets. PBC provides a quantitative characterization of the texture’s structuredness and directionality for browsing application, and the SRC characterizes the distribution of texture energy in di!erent subbands, and supports similarity retrieval. This representation is quite robust to illumination variations and compares favorably with other texture descriptors for similarity retrieval. Experimental results are provided. ( 2000 Elsevier Science B.V. All rights reserved.