Search

We introduce NeuroConstruct, a novel end-to-end application for the segmentation, registration, and visualization of brain volumes imaged using wide-field microscopy. NeuroConstruct offers a Segmentation Toolbox with various annotation helper functions that aid experts to effectively and precisely annotate micrometer resolution neurites. It also offers an automatic neurites segmentation using convolutional neuronal networks (CNN) trained by the Toolbox annotations and somas segmentation using thresholding. To visualize neurites in a given volume, NeuroConstruct offers a hybrid rendering by combining iso-surface rendering of high-confidence classified neurites, along with real-time rendering of raw volume using a 2D transfer function for voxel classification score versus voxel intensity value. For a complete reconstruction of the 3D neurites, we introduce a Registration Toolbox that provides automatic coarse-to-fine alignment of serially sectioned samples. The quantitative and qualitative analysis show that NeuroConstruct outperforms the state-of-the-art in all design aspects. NeuroConstruct was developed as a collaboration between computer scientists and neuroscientists, with an application to the study of cholinergic neurons, which are severely affected in Alzheimer's disease.

Extracting and analyzing crime patterns in big cities is a challenging spatiotemporal problem. The hardness of the problem is linked to two main factors, the sparse nature of the crime activity and its spread in large spatial areas. Sparseness hampers most time series (crime time series) comparison methods from working properly, while the handling of large urban areas tends to render the computational costs of such methods impractical. Visualizing different patterns hidden in crime time series data is another issue in this context, mainly due to the number of patterns that can show up in the time series analysis. In this article, we present a new methodology to deal with the issues above, enabling the analysis of spatiotemporal crime patterns in a street-level of detail. Our approach is made up of two main components designed to handle the spatial sparsity and spreading of crimes in large areas of the city. The first component relies on a stochastic mechanism from which one can visually analyze probable×intensive crime hotspots. Such analysis reveals important patterns that can not be observed in the typical intensity-based hotspot visualization. The second component builds upon a deep learning mechanism to embed crime time series in Cartesian space. From the embedding, one can identify spatial locations where the crime time series have similar behavior. The two components have been integrated into a web-based analytical tool called CriPAV (Crime Pattern Analysis and Visualization), which enables global as well as a street-level view of crime patterns. Developed in close collaboration with domain experts, CriPAV has been validated through a set of case studies with real crime data in São Paulo - Brazil. The provided experiments and case studies reveal the effectiveness of CriPAV in identifying patterns such as locations where crimes are not intense but highly probable to occur as well as locations that are far apart from each other but bear similar crime patterns.

Deep reinforcement learning (DRL) targets to train an autonomous agent to interact with a pre-defined environment and strives to achieve specific goals through deep neural networks (DNN). Recurrent neural network (RNN) based DRL has demonstrated superior performance, as RNNs can effectively capture the temporal evolution of the environment and respond with proper agent actions. However, apart from the outstanding performance, little is known about how RNNs understand the environment internally and what has been memorized over time. Revealing these details is extremely important for deep learning experts to understand and improve DRLs, which in contrast, is also challenging due to the complicated data transformations inside these models. In this article, we propose Deep Reinforcement Learning Interactive Visual Explorer (DRLIVE), a visual analytics system to effectively explore, interpret, and diagnose RNN-based DRLs. Having focused on DRL agents trained for different Atari games, DRLIVE accomplishes three tasks: game episode exploration, RNN hidden/cell state examination, and interactive model perturbation. Using the system, one can flexibly explore a DRL agent through interactive visualizations, discover interpretable RNN cells by prioritizing RNN hidden/cell states with a set of metrics, and further diagnose the DRL model by interactively perturbing its inputs. Through concrete studies with multiple deep learning experts, we validated the efficacy of DRLIVE.

Event sequences are central to the analysis of data in domains that range from biology and health, to logfile analysis and people's everyday behavior. Many visualization tools have been created for such data, but people are error-prone when asked to judge the similarity of event sequences with basic presentation methods. This article describes an experiment that investigates whether local and global alignment techniques improve people's performance when judging sequence similarity. Participants were divided into three groups (basic versus local versus global alignment), and each participant judged the similarity of 180 sets of pseudo-randomly generated sequences. Each set comprised a target, a correct choice and a wrong choice. After training, the global alignment group was more accurate than the local alignment group (98 versus 93 percent correct), with the basic group getting 95 percent correct. Participants' response times were primarily affected by the number of event types, the similarity of sequences (measured by the Levenshtein distance) and the edit types (nine combinations of deletion, insertion and substitution). In summary, global alignment is superior and people's performance could be further improved by choosing alignment parameters that explicitly penalize sequence mismatches.

Triangle meshes are used in many important shape-related applications including geometric modeling, animation production, system simulation, and visualization. However, these meshes are typically generated in raw form with several defects and poor-quality elements, obstructing them from practical application. Over the past decades, different surface remeshing techniques have been presented to improve these poor-quality meshes prior to the downstream utilization. A typical surface remeshing algorithm converts an input mesh into a higher quality mesh with consideration of given quality requirements as well as an acceptable approximation to the input mesh. In recent years, surface remeshing has gained significant attention from researchers and engineers, and several remeshing algorithms have been proposed. However, there has been no survey article on remeshing methods in general with a defined search strategy and article selection mechanism covering the recent approaches in surface remeshing domain with a good connection to classical approaches. In this article, we present a survey on surface remeshing techniques, classifying all collected articles in different categories and analyzing specific methods with their advantages, disadvantages, and possible future improvements. Following the systematic literature review methodology, we define step-by-step guidelines throughout the review process, including search strategy, literature inclusion/exclusion criteria, article quality assessment, and data extraction. With the aim of literature collection and classification based on data extraction, we summarized collected articles, considering the key remeshing objectives, the way the mesh quality is defined and improved, and the way their techniques are compared with other previous methods. Remeshing objectives are described by angle range control, feature preservation, error control, valence optimization, and remeshing compatibility. The metrics used in the literature for the evaluation of surface remeshing algorithms are discussed. Meshing techniques are compared with other related methods via a comprehensive table with indices of the method name, the remeshing challenge met and solved, the category the method belongs to, and the year of publication. We expect this survey to be a practical reference for surface remeshing in terms of literature classification, method analysis, and future prospects.

The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-loop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinement. Quantitative experiments show that our active learning approach can accurately extract meaningful visual concepts. More importantly, by identifying visual concepts that negatively affect model performance, we develop the corresponding data augmentation strategy that consistently improves model performance., Comment: 11 pages, 10 figures

A growing number of longitudinal cohort studies are generating data with extensive patient observations across multiple timepoints. Such data offers promising opportunities to better understand the progression of diseases. However, these observations are usually treated as general events in existing visual analysis tools. As a result, their capabilities in modeling disease progression are not fully utilized. To fill this gap, we designed and implemented ThreadStates, an interactive visual analytics tool for the exploration of longitudinal patient cohort data. The focus of ThreadStates is to identify the states of disease progression by learning from observation data in a human-in-the-loop manner. We propose a novel Glyph Matrix design and combine it with a scatter plot to enable seamless identification, observation, and refinement of states. The disease progression patterns are then revealed in terms of state transitions using Sankey-based visualizations. We employ sequence clustering techniques to find patient groups with distinctive progression patterns, and to reveal the association between disease progression and patient-level features. The design and development were driven by a requirement analysis and iteratively refined based on feedback from domain experts over the course of a 10-month design study. Case studies and expert interviews demonstrate that ThreadStates can successively summarize disease states, reveal disease progression, and compare patient groups.

Semantic segmentation is a critical component in autonomous driving and has to be thoroughly evaluated due to safety concerns. Deep neural network (DNN) based semantic segmentation models are widely used in autonomous driving. However, it is challenging to evaluate DNN-based models due to their black-box-like nature, and it is even more difficult to assess model performance for crucial objects, such as lost cargos and pedestrians, in autonomous driving applications. In this work, we propose VASS, a Visual Analytics approach to diagnosing and improving the accuracy and robustness of Semantic Segmentation models, especially for critical objects moving in various driving scenes. The key component of our approach is a context-aware spatial representation learning that extracts important spatial information of objects, such as position, size, and aspect ratio, with respect to given scene contexts. Based on this spatial representation, we first use it to create visual summarization to analyze models' performance. We then use it to guide the generation of adversarial examples to evaluate models' spatial robustness and obtain actionable insights. We demonstrate the effectiveness of VASS via two case studies of lost cargo detection and pedestrian detection in autonomous driving. For both cases, we show quantitative evaluation on the improvement of models' performance with actionable insights obtained from VASS.

We introduce a new online database of nucleic acid nanostructures for the field of DNA and RNA nanotechnology. The database implements an upload interface, searching and database browsing. Each deposited nanostructures includes an image of the nanostructure, design file, an optional 3D view, and additional metadata such as experimental data, protocol or literature reference. The database accepts nanostructures in any preferred format used by the uploader for the nanostructure design. We further provide a set of conversion tools that encourage design file conversion into common formats (oxDNA and PDB) that can be used for setting up simulations, interactive editing or 3D visualization. The aim of the repository is to provide to the DNA/RNA nanotechnology community a resource for sharing their designs for further reuse in other systems and also to function as an archive of the designs that have been achieved in the field so far. Nanobase.org is available at https://nanobase.org/.

Multimedia data play an important role in many areas such as business, medicine, and entertainment. Multimedia contents are often vulnerable to malicious interventions due to the advancement in multimedia innovations. It is, therefore, highly necessary to verify the integrity and authenticity of multimedia content through appropriate security measures. This increases the need to establish a range of watermarking schemes for several security purposes, including copyrights and authentication. Digital watermarking has proven an effective and influential approach for 3D models among various other authentication strategies. In this study, an in-depth review of different types of 3D model representations, watermarking techniques in spectral and spatial domains along with most common attacks that can be applied to them are presented. The paper concludes with a set of observations along with some of the research issues and challenges which are needed to be studied further.

Bridging Centrality (BriCe) is a popular measure that combines the Betweenness centrality and Bridging coefficient metrics to characterize nodes acting as a bridge among clusters. However, there were no implementations of the BriCe plugin that can be readily used in the GEPHI software or any other software dedicated to graph-based studies. In this paper, we present the BriCe plugin for GEPHI. It is available as a third-party functionality from the native GEPHI interface as a handy plugin to add; hence, no additional download and installation process is necessary. The BriCe plugin for GEPHI is open-source, and one can access the code through the GEPHI GitHub repository. As a use case of the BriCe plugin, we analyzed the genome of Mycobacterium tuberculosis H37Rv to identify biological explanations on why some proteins were ranked with top BriCe values? For instance, we were able to formulate a new hypothesis combining the predicted sub cellular localization and high BriCe values concerning lipopolysaccharides (LPS) exportation. Our hypothesis provides a possible link among proteins of a glycosyltransferase group and the type VII Secretion System. The Bridging Centrality plugin for GEPHI is an easy to use tool for analyzing complex graphs and draw novel insights from graphical data.

Video quality assessment (VQA) algorithms play an significant role in many fields of video analysis. To improve the accuracy of video quality evaluation, many scholars start by imitating viewers’ subjective feelings to make the algorithm more suitable for the subjective feelings of the video viewers. On the other hand, many scholars are also working to reduce some unnecessary time costs caused by frame-by-frame analysis. To achieve this goal, it also caused a lot of unnecessary time costs. To solve these problems, this paper proposes an improved algorithm based on danmaku analysis. First of all, through the analysis of eye movement data, we find that the concentrated expression of the subjective emotion of the video viewers contains a strong direct relationship with the quality of the video, and the danmaku is an element of such subjective emotion. Then we improve the existing video quality assessment algorithm, analyze the significant role of danmaku in subjective emotion, extract specific keyframes, and get the corresponding objective score as the result. The experimental results show that the algorithm improves the efficiency of time optimization, and from the Pearson correlation coefficient (PCC), the results are more fit for viewers’ subjective feelings.

3D shape completion from single-view scan is an important task for follow-up applications such as recognition and segmentation, but it is challenging due to the critical sparsity and structural incompleteness of single-view point clouds. In this paper, a three-stage generative network (TSGN) is proposed for single-view point cloud completion, which generates fine-grained dense point clouds step by step and effectively overcomes the ubiquitous problem—the imbalance between general and individual characteristics. In the first stage, an encoder–decoder network consumes a partial point cloud and generates a rough sparse point cloud inferring the complete geometric shape. Then, a bi-channel residual network is designed to refine the preliminary result with assistance of the original partial input. A local-based folding network is introduced in the last stage to extract local context information from the revised result and build a dense point cloud with finer-grained details. Experiments on ShapeNet dataset and KITTI dataset validate the effectiveness of TSGN. The results on ShapeNet demonstrate the competitive performance on both CD and EMD.

The objectives of this research were, (1) to develop a prototype of multimedia learning in Computer Graphics course, (2) to test the validity, practicality, and effectiveness of learning multimedia developed in Computer Graphics course. This research was a Research and Development (R&D) design with ADDIE model include thirty students as a sample. The procedure of this research until implementation stage. Instrument of the research was a questionnaire. Data of need analysis obtained based on the review of the exist curriculum, the characteristics of the materials and students need. The students need media that is not only in the conventional form of the file that just containing text but more than it, the media which is able to provide experience directly in the multimedia form that containing images, text, animation, and videos. The multimedia design of computer graphics learning is specifically designed from the structure, contents, menu placement, color selection, and so on to make it easier for lecturers and students. The results of this research are: (1) The results of the analysis of the data validity of the media are in the very valid category, this is obtained from the expert validation of material and media. (2) The results of the evaluation of the students' trials and the lecturers' responses to this multimedia are very practical. (3) this learning multimedia was effective as seen from the recapitulation of the pre-test and post-test learning outcomes assessment showing a significant increase in student learning scores.

3D scanning devices are traditionally employed in reverse engineering tasks that can be carried out semi-automatically, with user assistance. However, their application in manufacturing process control requires automatic point cloud segmentation and extraction of geometric primitives. In this paper, we propose a method for automatic recognition of planes and cylinders (most frequently encountered geometric primitives in mechanical engineering) from unstructured point clouds. The method is based on the scatter of data during least squares fitting of second order surfaces. It consists of three phases and the first phase represents automatic point cloud segmentation. The second phase deals with merging of over-segmented regions and surfaces parameters estimation, whereas the final phase provides extraction of recognized geometric primitives. The method is experimentally verified using three real-world case studies, and its performances are compared with two state of the art recognition algorithms. The results have shown that the proposed method outperforms alternative approaches in terms of appropriately recognized planes and cylinders without surface type confusion, as well as when the recognition of non-existent primitives is considered. In addition, the method determines surfaces parameters with high accuracy.

The task of video captioning is to generate a video description corresponding to the video content, so there are stringent requirements for the extraction of fine-grained video features and the language processing of tag text. A new method using global control of the text and local strengthening during training is proposed in this paper. In this method, the context can be referred to when the training generates text. In addition, more attention is given to important words in the text, such as nouns and predicate verbs, and this approach greatly improves the recognition of objects and provides more accurate prediction of actions in the video. Moreover, in this paper, the authors adopt 2D and 3D multimodal feature extraction for the process of video feature extraction. Better results are achieved by the fine-grained feature capture of global attention and the fusion of bidirectional time flow. The method in this paper obtains good results on both the MSR-VTT and MSVD datasets.

Due to the low computational complexity, the two-phase test sample representation method shows outstanding advantages in the field of real-time face recognition. However, its first phase does not fully consider the imbalance of the determined K-nearest training samples. This defect is not well reflected and will seriously affect the recognition accuracy of the second phase. In this paper, we explore the above issues and propose to incorporate the unselected training samples into the modeling process. The proposed method not only allows the unselected samples to perform the final representation but also makes the selected nearest training samples play a more significant role than the unselected ones in classification. It is efficiently solved with an analytical solution. The rationales and probability interpretation of the proposed method are presented to further guarantee its rationality and effectiveness. Extensive experiments on diverse face databases are conducted to demonstrate the superior recognition performance in comparison with other competing classifiers.

Efficient visualization helps researchers obtain valuable mechanistic insights and present interesting results with regard to the functional enrichment analysis of omics data. However, the functions of existing published tools used to implement relevant visualization are neither sufficiently comprehensive nor easily accessible. Most of these tools require users to have professional programming skills. This study alleviates this issue by proposing EnrichVisBox, a web application developed for integrative and versatile data visualization, including bubble plots, UpSet plots, polar bar plots, rectangle plots, ridgeline plots, network plots, and variant chord plots. Specifically, scientists can use these insightful plots to conveniently present functional enrichment analysis results of omics data with a simple mouse click through a user-friendly interface.

The massive use of multimedia technologies has enabled the exploration of information in many data such as texts, audio, videos, and images. Computational methods are being developed for several purposes such as monitoring, security, business, and even health through the automatic diagnosis of diseases by medical images. Among these diseases, we have melanoma skin cancer. Melanoma is a skin cancer that causes a large number of fatalities worldwide. Several methods for the automatic diagnosis of melanoma in dermoscopic images have been developed. For these methods to be more efficient, it is essential to isolate the lesion region. This study used a melanoma segmentation method based on U-net and LinkNet deep learning networks combined with transfer learning and fine-tuning techniques. Additionally, we evaluate the model’s ability to learn to segment the disease or just the dataset by combining datasets. The experiments were carried out in three datasets (PH2, ISIC 2018, and DermIS) and obtained promising results, emphasizing the U-net that obtained an average Dice of 0.923 in the PH2 dataset, Dice = 0.893 in ISIC 2018, and Dice = 0.879 in the DermIS dataset.

Pedestrian detection using deep convolutional neural networks (DCNNs) has made a breakthrough in the last few years and researchers have proposed different DCNN architectures to detect pedestrians more accurately. Most of these architectures have a backbone based on previous state-of-the-art architectures for classification tasks and just tried to adapt them for their detection task. They are improving their performance with some heuristics, trial and error techniques, and sometimes with grid search on a space of various architectures. However, there is no research in which, firstly, the visual detection system of human has been studied, and then tried to propose a backbone architecture based on that. In this paper, we first review the state-of-the-art methods and then, having a preliminary on visual detection system in the human brain and finally, propose our architecture based on that. The intuition behind our idea can justify the evolutionary course of detection architectures from the first fully convolutional neural networks (FCNNs), like Faster R-CNN, to the modern state-of-the-art methods nowadays and give us a better understanding of why some architectures are superior to the others. The advantage of our idea is that it can be applied to most of the existing architectures with some manipulations, although it is much easier on some methods than others. We have implemented our idea based on an anchor-free method called CSP and could achieve better performance on Caltech-USA and INRIA, which are two of the most popular pedestrian detection datasets.

Using the design method proposed by Kunii, this study mathematically explains the achievement of Kublai of merging two different societies to build a globalized nation after conquering China. The considered design method, called incrementally modular abstraction hierarchy, consists of seven levels, from the most abstract, homotopy, to the most concrete, computer graphics. Moreover, it allows a flexible design while moving up and down its hierarchy. Using this method, we attempt to restore the concepts of Kublai, aiming to build a globalized state by combining nomadic and agricultural cultures. Specifically, we examine the use of two capitals to resolve the problem of different lifestyles of the Mongolian Khan, who moved between the summer and winter camps of the nomads, and the Chinese Emperor, who lived in a fixed palace of the agricultural people. By incorporating the category theory into Kunii’s method, we discuss the adjunction of the Mongolian 1000-household social and military system based on decimal numbers with the Chinese bureaucratic system. The inclusion of the category theory in his original method allows visualization by programming, making the design more versatile and flexible.

Realistic 3D indoor scene datasets have enabled significant recent progress in computer vision, scene understanding, autonomous navigation, and 3D reconstruction. But the scale, diversity, and customizability of existing datasets is limited, and it is time-consuming and expensive to scan and annotate more. Fortunately, combinatorics is on our side: there are enough individual rooms in existing 3D scene datasets, if there was but a way to recombine them into new layouts. In this paper, we propose the task of generating novel 3D floor plans from existing 3D rooms. We identify three sub-tasks of this problem: generation of 2D layout, retrieval of compatible 3D rooms, and deformation of 3D rooms to fit the layout. We then discuss different strategies for solving the problem, and design two representative pipelines: one uses available 2D floor plans to guide selection and deformation of 3D rooms; the other learns to retrieve a set of compatible 3D rooms and combine them into novel layouts. We design a set of metrics that evaluate the generated results with respect to each of the three subtasks and show that different methods trade off performance on these subtasks. Finally, we survey downstream tasks that benefit from generated 3D scenes and discuss strategies in selecting the methods most appropriate for the demands of these tasks., Symposium on Geometry Processing (SGP) 2021

With the growth of various types of media data (e.g., text, image, video, and audio), fine-grained cross-media retrieval which aims to provide flexible and accurate query service has attracted significant attention. Different from traditional keyword-based retrieval, the queries and results in fine-grained cross-media retrieval might be different types. In this work, we demonstrate that the quality of retrieval results can be further improved by additionally considering the media-specified information. Besides, the process of feature extraction should be different for queries with various media types. To this end, we propose a novel network architecture, namely Double Branch Fine-grained Cross-media Net (DBFC-Net), which is the first work that can use the media-specific information to construct the common features by a uniform framework. Furthermore, we devise an effective distance metric (cosine+) for fine-grained cross-media retrieval. Compared with commonly-used metrics (e.g., cosine function), our proposed cosine+ metric is well adaptive to handle fine-grained retrieval scenarios. Extensive experiments and ablation studies on publicly available datasets demonstrate the effectiveness of our proposed approach.

Face super-resolution aims to recover high-resolution face images with accurate geometric structures. Most of the conventional super-resolution methods are trained on paired data that is difficult to obtain in the real-world setting. Besides, these methods do not fully utilize facial prior knowledge for face super-resolution. To tackle these problems, we propose an end-to-end unsupervised face super-resolution network to super-resolve low-resolution face images. We propose a gradient enhancement branch and a semantic guidance mechanism. Specifically, the gradient enhancement branch reconstructs high-resolution gradient maps, under the restriction of two proposed gradient losses. Then the super-resolution network integrates features in both image and gradient space to super-resolve face images with geometric structure preservation. Moreover, the proposed semantic guidance mechanism, including a semantic-adaptive sharpen module and a semantic-guided discriminator, can reconstruct sharp edges and improve local details in different facial regions adaptively, under the guidance of semantic parsing maps. Qualitative and quantitative experiments demonstrate that our proposed method can reconstruct high-resolution face images with sharp edges and photo-realistic details, outperforming the state-of-the-art methods.

The depth perception of human visual system is divergent between virtual and real space; this depth discrepancy affects the spatial judgment of the user in a virtual space, which means the user cannot precisely locate their self-position in a virtual space. Existing localization methods ignore the depth discrepancy and only concentrate on increasing location accuracy in real space. Thus, the discrepancy always exists in virtual space, which induces visual discomfort. In this paper, a localization method based on depth perception is proposed to measure the self-position of the user in a virtual environment. Using binocular gaze tracking, this method estimates perceived depth and constructs an eye matrix by measuring gaze convergence on a target. Comparing the eye matrix and camera matrix, the method can automatically calculate the actual depth of the viewed target. Then, the difference between the actual depth and the perceived depth can be explicitly estimated without markers. The position of the virtual camera is compensated by the depth difference to obtain perceptual self-position. Furthermore, a virtual reality system is redesigned by adjusting the virtual camera position. The redesigned system makes users feel that the distance (from the user to an object) is the same in virtual and real space. Experimental results demonstrate that the redesigned system can improve the user’s visual experiences, which validate the superiority of the proposed localization method.

Image forgeries created using easy-to-use image editing tools like Adobe Photoshop have become prime sources of fake news, and are often used in malevolent ways in politics, courtrooms, and in scientific publishing as well. To reduce the problems caused by manipulated images in these domains, it is important to have reliable and fast mechanisms in place for verifying their authenticity before their consumption. Automated detection of image forgeries, is, therefore an ongoing challenge for the research community. One of a commonly used image forgeries is Copy-Move Forgery (CMF), where a region of an image is copied and pasted to some other location within the same image to alter its contents. To detect copy-move forgeries, several Copy Move Forgery Detection (CMFD) techniques have been proposed in the past. However, existing techniques are highly sensitive to post-processing attacks such as noise and compression. In this paper, we propose a new CMFD method in which Cellular Automata (CA) is used for feature extraction. The method works by dividing the input image into overlapping blocks and extracting features from these blocks using the CA inversion procedure. Then, the repeated regions are identified by matching the set of extracted feature vectors. The experimental results show that the detection under noise and compression attacks is accurate compared to many existing methods. As such, the proposed approach may be useful for a variety of CMFD scenarios and may have implications for researchers in related fields.

Generating computer graphics (CG) rendered synthetic images has been widely used to create simulation environments for robotics/autonomous driving and generate labeled data. Yet, the problem of training models purely with synthetic data remains challenging due to the considerable domain gaps caused by current limitations on rendering. In this paper, we propose a simple yet effective domain adaptation framework towards closing such gap at image level. Unlike many GAN-based approaches, our method aims to match the covariance of the universal feature embeddings across domains, making the adaptation a fast, convenient step and avoiding the need for potentially difficult GAN training. To align domains more precisely, we further propose a conditional covariance matching framework which iteratively estimates semantic segmentation regions and conditionally matches the class-wise feature covariance given the segmentation regions. We demonstrate that both tasks can mutually refine and considerably improve each other, leading to state-of-the-art domain adaptation results. Extensive experiments under multiple synthetic-to-real settings show that our approach exceeds the performance of latest domain adaptation approaches. In addition, we offer a quantitative analysis where our framework shows considerable reduction in Frechet Inception distance between source and target domains, demonstrating the effectiveness of this work in bridging the synthetic-to-real domain gap.

Problems of the multiple testing of hypotheses on the significance of observations hold an important place in applied statistics and are used in a wide variety of areas such as genetics, biology, astronomy, and computer graphics. These problems arise especially ehen processing multidimensional data in order to identify significant features and remove insignificant (noise) features. Modern studies devoted to this topic classify different means of filtering based on sparse representation of experimental data. A popular approach to constructing statistical estimates of patterns in experimental data includes the multiple testing of hypotheses on the significance of observations. In this work, we consider the Benjamini–Hochberg algorithm for multiple hypothesis testing when selecting the FDR threshold for sparse data. A theorem is proved by estimating the rate of convergence of the risk estimator to the normal distribution.

Precisely editing user specified facial attributes has wide applications in areas such as virtual makeup, face aging, facial expression transfer, face synthesis. However, it is difficult to explicitly control individual facial attribute due to the gap between high level semantics in human perception and feature vectors in latent space. In this paper, a semantic disentanglement algorithm interpreting the latent space of GAN is proposed, which can be employed to extract attribute control vector adaptive to individual face. By adjusting the coefficient of extracted control vector, variation of single attribute is realized. Then, comprehensive modification effect of facial attributes is obtained through the superposition of control vector. Classification and content loss functions are introduced to limit modification occurs to the specified attribute without affecting the other attributes. As a result, precise editing control is realized.

Character inpainting is an attractive and challenging task, especially for Chinese calligraphy characters with complex structures and styles. The diversity of Chinese calligraphy styles has created its unique artistic beauty, but specific style features will lead to obvious differences in style and stroke details while recovering. Most current methods are restricted to recover specific characters already present in the training set and retrain the model when recovering characters of new styles. Moreover, these methods are based on edge recovery, which requires the location of the masked area. In this paper, we propose a novel structure-guided generative framework guided by prototype character, which can not only adapt to multiple style fonts but also overall recover glyph structure and strokes without masked information by inferring the style representation. In this case, our method can recover new style characters, which is the first attempt for character inpainting without parameter retraining. Experimental results demonstrate that our method has generalizability and superiority in most application scenarios, compared with several state-of-the-art character inpainting methods.

The article discusses the formation of professional competencies of students through computer graphics. It also describes the main goals and objectives of teaching computer graphics, the methodological basis of image creation, processing using the basics of software and hardware, engineering graphics and animation.

Computational approaches for predicting drug-target interactions (DTIs) play an important role in drug discovery since conventional screening experiments are time-consuming and expensive. In this study, we proposed end-to-end representation learning of a graph neural network with an attention mechanism and an attentive bidirectional long short-term memory (BiLSTM) to predict DTIs. For efficient training, we introduced a bidirectional encoder representations from transformers (BERT) pretrained method to extract substructure features from protein sequences and a local breadth-first search (BFS) to learn subgraph information from molecular graphs. Integrating both models, we developed a DTI prediction system. As a result, the proposed method achieved high performances with increases of 2.4% and 9.4% for AUC and recall, respectively, on unbalanced datasets compared with other methods. Extensive experiments showed that our model can relatively screen potential drugs for specific protein. Furthermore, visualizing the attention weights provides biological insight.

Locomotion is a fundamental interaction element allowing navigation inside the virtual environment, and the walking-in-place (WIP) techniques have been actively developed as a balanced compromise between naturalness and efficiency. One popular method to implement the WIP technique was to use a low-cost, easy to set up, and markerless Kinect, but required integration of multiple sensors or covered limited directions due to the poor tracking capability when facing non-frontal sides of the user. This study aimed to propose a WIP technique for omnidirectional VR locomotion based on a single RGB camera, utilizing an open-source 2D human pose estimation system called OpenPose. Three WIP techniques (existing Kinect-based technique, proposed Kinect-based technique, and proposed OpenPose-based technique) were compared in terms of variation of virtual walking speed and subjective evaluation through a user study with walking tasks in different directions. Experimental results showed that the proposed OpenPose-based technique performed comparably when the user faced the front of the camera, but it induced lower variation of virtual walking speed and higher subjective evaluation ratings at non-forward directions compared to other techniques. The proposed OpenPose-based WIP technique can be used in VR applications to provide a fully unobstructed VR locomotion experience. It can achieve stable WIP-based omnidirectional VR locomotion through a single low-cost easily accessible RGB camera, without the need for additional sensors, and at the same time, both hands are free for other interactions.

In recent times, active research is going on for bridging the gap between computer vision and natural language. In this paper, we attempt to address the problem of Hindi video captioning. In a linguistically diverse country like India, it is important to provide a means which can help in understanding the visual entities in native languages. In this work, we employ a hybrid attention mechanism by extending the soft temporal attention mechanism with a semantic attention to make the system able to decide when to focus on visual context vector and semantic input. The visual context vector of the input video is extracted using 3D convolutional neural network (3D CNN) and a Long Short-Term Memory (LSTM) recurrent network with attention module is used for decoding the encoded context vector. We experimented on a dataset built in-house for Hindi video captioning by translating $$MSR-VTT$$ dataset followed by post-editing. Our system achieves 0.369 CIDEr score and 0.393 METEOR score and outperformed other baseline models including RMN (Reasoning Module Networks)-based model.

Early identification of crop disease can aid the farmers to take timely precautions and countermeasures for its removal. In this paper, a real-time system to identify the type of disease present in a crop based on leaf images using machine learning is proposed. A deep convolutional neural network architecture is proposed to classify the crop disease, and a single shot detector is used for identification and localization of the leaf. These models are deployed on an embedded hardware, Nvidia Jetson TX1, for real-time in-field plant disease detection and identification. The disease classification accuracy achieved is around 96.88%, and the classification results are compared with existing convolutional neural network architectures. Also, the high success rate of the proposed system in the actual field test makes the proposed system a completely deployable system.

Fashion is a form of self-expression that permits us to manifest our personality and identity with more confidence. Visual fashion clothing analysis has attracted researchers who have sought to pioneer deep learning concepts. In this work, we introduce a semi-supervised multi-task learning approach intending to attain clothing category classification and attribute prediction. For intensifying semi-supervised fashion clothes analysis, we embrace a teacher–student (T–S) pair model that utilises weighted loss minimisation while sharing knowledge between teacher and student. Our focus in this work is on strengthening the feature representation by simultaneous learning of labelled and unlabelled samples that avoids additional training for unlabelled samples. As a result, our approach involves in gaining beneficiary performance by making use of semi-supervised learning in fashion clothing analysis. We evaluated the proposed approach on the large-scale DeepFashion-C dataset and the combined unlabelled dataset obtained from six publicly available datasets. Experimental results show that the proposed paired architecture involving deep neural networks is comparable to state-of-the-art techniques in fashion clothing analysis.

The contribution of this paper is toward three directions, namely (a) in identifying the advantages and disadvantages of the current state-of-the-art methods that allow laymen in programming to author VR experiences; (b) in examining how easily non-experts in programming can author VR experiences by analysing the responses from several testers toward easiness and usability when using the proposed authoring tool; and (c) in treating a serious disadvantage of state-of-the-art methods, namely the low quality in graphics by proposing a novel methodology for transpiling web-based formats such as three.js into high performance runtime formats like Unity3D. The proposed authoring tool is a plugin for WordPress that exploits its interfaces and database for providing a 3D editor suitable for authoring VR experiences. It embeds the proposed methodology which achieves an one-to-one matching between three.js, WordPress and Unity3D entities to achieve transpiling. Evaluation results indicate the positive adoption of non-experts in programming, but there is still several improvements to be made.

Virtual reality (VR) is a promising tool and is increasingly used in many different fields, in which virtual walking can be generalized through detailed modeling of the physical environment such as in sports science, medicine and furthermore. However, the visualization of a virtual environment using a head-mounted display (HMD) differs compared to reality, and it is still not clear whether the visual perception works equally within VR. The purpose of the current study is to compare the spatial orientation between real world (RW) and VR. Therefore, the participants had to walk blindfolded to different placed objects in a real and virtual environment, which did not differ in physical properties. They were equipped with passive markers to track the position of the back of their hand, which was used to specify each object’s location. The first task was to walk blindfolded from one starting position to different placed sport-specific objects requiring different degrees of rotation after observing them for 15 s (0°, 45°, 180°, and 225°). The three-way ANOVA with repeated measurements indicated no significant difference between RW and VR within the different degrees of rotation (p > 0.05). In addition, the participants were asked to walk blindfolded three times from a new starting position to two objects, which were ordered differently during the conditions. Except for one case, no significant differences in the pathways between RW and VR were found (p > 0.05). This study supports that the use of VR ensures similar behavior of the participants compared to real-world interactions and its authorization of use.

This paper addresses the evaluation of algorithms reconstructing a watertight surface from a point cloud acquired on an open scene. The objective is to set a rigorous protocol measuring the quality of the reconstruction and to propose a quality metric that is informative with respect to the various qualities that such an algorithm should have, and in particular its capacity to interpolate and extrapolate accurately. Our approach aims at being more informative and rigorous than previous works on this topic. In addition, we use publicly available data and our implementation is open-source. We argue that a rigorous evaluation of surface reconstruction of open scenes needs to be performed on synthetic data where a perfect continuous ground truth surface is available, so we developed our own LiDAR simulator of which we give a description in the present paper.

We present a data-driven algorithm for generating gaits of virtual characters with varying dominance traits. Our formulation utilizes a user study to establish a data-driven dominance mapping between gaits and dominance labels. We use our dominance mapping to generate walking gaits for virtual characters that exhibit a variety of dominance traits while interacting with the user. Furthermore, we extract gait features based on known criteria in visual perception and psychology literature that can be used to identify the dominance levels of any walking gait. We validate our mapping and the perceived dominance traits by a second user study in an immersive virtual environment. Our gait dominance classification algorithm can classify the dominance traits of gaits with ˜73 percent accuracy. We also present an application of our approach that simulates interpersonal relationships between virtual characters. To the best of our knowledge, ours is the first practical approach to classifying gait dominance and generate dominance traits in virtual characters.

The identification of interesting patterns and relationships is essential to exploratory data analysis. This becomes increasingly difficult in high dimensional datasets. While dimensionality reduction techniques can be utilized to reduce the analysis space, these may unintentionally bury key dimensions within a larger grouping and obfuscate meaningful patterns. With this work we introduce DimLift , a novel visual analysis method for creating and interacting with dimensional bundles . Generated through an iterative dimensionality reduction or user-driven approach, dimensional bundles are expressive groups of dimensions that contribute similarly to the variance of a dataset. Interactive exploration and reconstruction methods via a layered parallel coordinates plot allow users to lift interesting and subtle relationships to the surface, even in complex scenarios of missing and mixed data types. We exemplify the power of this technique in an expert case study on clinical cohort data alongside two additional case examples from nutrition and ecology. acceptedVersion

For a long time, Graphic design was specialized in the production of printed materials for example magazines, newspapers, books, flyers, advertising posters …etc. However, the development in the field of information technology added a lot to graphic design, it came with a standalone branch called "Multimedia", which deals with computer graphics - without the need to print them - and forms them in a dynamic framework such as movies, With the ability to use sound effects, and the most popular programs dealing with this type of design is "3d Max, Maya, Macromedia Director, Adobe Flash, & Multimedia Builder". Even hand-drawn animation became drawing and moving by computer in a much less time than before. There are different types of animations but not limited to 2D animations, 3D animations, Augmented Reality, Virtual Reality …etc. The importance of the research is due to the fact that in our time, it has become of great importance to deeply use the concept and methods of graphic processing of advertising campaigns using augmented reality and three-dimensional hologram imaging and give a special character in those animated advertising campaigns. Change graphic designers' thinking about graphic designs using hologram imaging and mix it with augmented reality. It is in our modern age that it has become a phenomenon in the field of graphic design, and it must be studied with an examination to find out its results on modern thought, especially in the field of advertising and graphic advertising campaigns.

The main goal of the super-resolution (SR) is to restore a visually pleasing high-resolution image using a low-resolution image of video sequence. The higher resolution image is composed of higher pixel density with fine and precise details as compared with the low-resolution images or video. The majority of the applications, such as video surveillance, ultra-high-definition TV, low-resolution face recognition and remote sensing imaging are based on super-resolutions. Thus benefiting from the broader spectrun of these applications, the super-resolution has attracted massive interest form both acemedia and industry. And currently, a most active research field in today’s era.

In the task of semantic segmentation, researchers often use self-attention module to capture long-range contextual information. These methods are often effective. However, the use of the self-attention module will cause a problem that cannot be ignored, that is, the huge consumption of computing resources. Therefore, how to reduce the resource consumption of the self-attention module under the premise of ensuring performance is a very meaningful research topic. In this paper, we propose a Sparse Attention Model combined with a powerful multi-task feature extraction network for semantic segmentation. Compared with the classic self-attention model, our Sparse Attention Model does not calculate the inner product between pairs of all vectors. Instead, we first sparse the feature block Query and the feature block Key defined in self-attention module through the credit matrix generated by the pre-output. Then, we perform similarity modeling on the two sparse feature blocks. Meanwhile, to ensure that the vectors in Query could capture dense contextual information, we design a Class Attention Module and embed it into Sparse Attention Module. Note that, compared with Dual Attention Network for scene segmentation, our attention module greatly reduces the consumption of computing resources while ensuring the accuracy. Furthermore, in the stage of feature extraction, the use of downsampling will cause serious loss of detailed information and affect the segmentation performance of the network, so we adopt a multi-task feature extraction network. It learns semantic features and edge features in parallel, and we feed the learned edge features into the deep layer of the network to help restore detailed information for capturing high-quality semantic features. We do not use pure concatenation. Instead, we extract the edge features related to each channel by element-wise multiplication before concatenation. Finally, we conduct experiments on three datasets: Cityscapes, PASCAL VOC2012 and ADE20K, and obtain competitive results.

The simulation of subtractive manufacturing processes has a long history in engineering. Corresponding predictions are utilized for planning, validation and optimization, e.g., of CNC-machining processes. With the up-rise of flexible robotic machining and the advancements of computational and algorithmic capability, the simulation of the coupled machine-process behaviour for complex machining processes and large workpieces is within reach. These simulations require fast material removal predictions and analysis with high spatial resolution for multi-axis operations. Within this contribution, we propose to leverage voxel-based concepts introduced in the computer graphics industry to accelerate material removal simulations. Corresponding schemes are well suited for massive parallelization. By leveraging the computational power offered by modern graphics hardware, the computational performance of high spatial accuracy volumetric voxel-based algorithms is further improved. They now allow for very fast and accurate volume removal simulation and analysis of machining processes. Within this paper, a detailed description of the data structures and algorithms is provided along a detailed benchmark for common machining operations.

Three-dimensional (3D) reconstruction of shapes is an important research topic in the fields of computer vision, computer graphics, pattern recognition, and virtual reality. Existing 3D reconstruction methods usually suffer from two bottlenecks: (1) they involve multiple manually designed states which can lead to cumulative errors, but can hardly learn semantic features of 3D shapes automatically; (2) they depend heavily on the content and quality of images, as well as precisely calibrated cameras. As a result, it is difficult to improve the reconstruction accuracy of those methods. 3D reconstruction methods based on deep learning overcome both of these bottlenecks by automatically learning semantic features of 3D shapes from low-quality images using deep networks. However, while these methods have various architectures, in-depth analysis and comparisons of them are unavailable so far. We present a comprehensive survey of 3D reconstruction methods based on deep learning. First, based on different deep learning model architectures, we divide 3D reconstruction methods based on deep learning into four types, recurrent neural network, deep autoencoder, generative adversarial network, and convolutional neural network based methods, and analyze the corresponding methodologies carefully. Second, we investigate four representative databases that are commonly used by the above methods in detail. Third, we give a comprehensive comparison of 3D reconstruction methods based on deep learning, which consists of the results of different methods with respect to the same database, the results of each method with respect to different databases, and the robustness of each method with respect to the number of views. Finally, we discuss future development of 3D reconstruction methods based on deep learning.

A primary affordance of virtual reality (VR) headsets is to give the user spatial presence or the illusion of being in the virtual environment. Although considerable research connects VR to spatial presence, spatial awareness, and spatial ability, little is known about how users develop spatial presence in VR learning environments. This study addresses that gap by exploring spatial presence experience construction in a VR educational game and investigating whether users’ knowledge, game experience, and VR experience impact the establishment of spatial presence. In this study, 56 high school students played an immersive 3D VR cell biology game where players search for clues within a virtual cell to diagnose the cell. Findings suggest that players’ perceptions of spatial presence are linked to how they allocate their attention during the game, their level of interest in cellular biology, and their visual-spatial acuity, but are not linked to their game experience, VR experience, or prior knowledge of the content area. These results indicate that well-scaffolded, engaging virtual environments can foster spatial presence among users, regardless of prior knowledge or experience, and gives practitioners clues about how to design VR learning environments.

Face views are particularly important in person-to-person communication. Differenes between the camera location and the face orientation can result in undesirable facial appearances of the participants during video conferencing. This phenomenon is particularly noticeable when using devices where the front-facing camera is placed in unconventional locations such as below the display or within the keyboard. In this paper, we take a video stream from a single RGB camera as input, and generate a video stream that emulates the view from a virtual camera at a designated location. The most challenging issue in this problem is that the corrected view often needs out-of-plane head rotations. To address this challenge, we reconstruct the 3D face shape and re-render it into synthesized frames according to the virtual camera location. To output the corrected video stream with natural appearance in real time, we propose several novel techniques including accurate eyebrow reconstruction, high-quality blending between the corrected face image and background, and template-based 3D reconstruction of glasses. Our system works well for different lighting conditions and skin tones, and can handle users wearing glasses. Extensive experiments and user studies demonstrate that our method provides high-quality results.

Objective Clinical trials are an essential part of the effort to find safe and effective prevention and treatment for COVID-19. Given the rapid growth of COVID-19 clinical trials, there is an urgent need for a better clinical trial information retrieval tool that supports searching by specifying criteria, including both eligibility criteria and structured trial information. Materials and Methods We built a linked graph for registered COVID-19 clinical trials: the COVID-19 Trial Graph, to facilitate retrieval of clinical trials. Natural language processing tools were leveraged to extract and normalize the clinical trial information from both their eligibility criteria free texts and structured information from ClinicalTrials.gov. We linked the extracted data using the COVID-19 Trial Graph and imported it to a graph database, which supports both querying and visualization. We evaluated trial graph using case queries and graph embedding. Results The graph currently (as of October 5, 2020) contains 3392 registered COVID-19 clinical trials, with 17 480 nodes and 65 236 relationships. Manual evaluation of case queries found high precision and recall scores on retrieving relevant clinical trials searching from both eligibility criteria and trial-structured information. We observed clustering in clinical trials via graph embedding, which also showed superiority over the baseline (0.870 vs 0.820) in evaluating whether a trial can complete its recruitment successfully. Conclusions The COVID-19 Trial Graph is a novel representation of clinical trials that allows diverse search queries and provides a graph-based visualization of COVID-19 clinical trials. High-dimensional vectors mapped by graph embedding for clinical trials would be potentially beneficial for many downstream applications, such as trial end recruitment status prediction and trial similarity comparison. Our methodology also is generalizable to other clinical trials.

Human recognition systems based on biometrics are much in demand due to increasing concerns of security and privacy. The human ear is unique and useful for recognition. It offers numerous advantages over popular biometrics traits face, iris, and fingerprints. A lot of work has been attributed to ear biometric, and the existing methods have achieved remarkable success over constrained databases. However, in unconstrained environment, a significant level of difficulty is observed as the images experience various challenges. In this paper, we first have provided a comprehensive survey on ear biometric using a novel taxonomy. The survey includes in-depth details of databases, performance evaluation parameters, and existing approaches. We have introduced a new database, NITJEW, for evaluation of unconstrained ear detection and recognition. A modified deep learning models Faster-RCNN and VGG-19 are used for ear detection and ear recognition tasks, respectively. The benchmark comparative assessment of our database is performed with six existing popular databases. Lastly, we have provided insight into open-ended research problems worth examining in the near future. We hope that our work will be a stepping stone for new researchers in ear biometrics and helpful for further development.