Search

Capturing long-range dependencies during feature extraction is crucial for video-based person re-identification (re-id) since it would help to tackle many challenging problems such as occlusion and dramatic pose variation. Moreover, capturing subtle differences, such as bags and glasses, is indispensable to distinguish similar pedestrians. In this paper, we propose a novel and efficacious Statistic Attention (SA) block which can capture both the long-range dependencies and subtle differences. SA block leverages high-order statistics of feature maps, which contain both long-range and high-order information. By modeling relations with these statistics, SA block can explicitly capture long-range dependencies with less time complexity. In addition, high-order statistics usually concentrate on details of feature maps and can perceive the subtle differences between pedestrians. In this way, SA block is capable of discriminating pedestrians with subtle differences. Furthermore, this lightweight block can be conveniently inserted into existing deep neural networks at any depth to form Statistic Attention Network (SANet). To evaluate its performance, we conduct extensive experiments on two challenging video re-id datasets, showing that our SANet outperforms the state-of-the-art methods. Furthermore, to show the generalizability of SANet, we evaluate it on three image re-id datasets and two more general image classification datasets, including ImageNet. The source code is available at http://vipl.ict.ac.cn/resources/codes/code/SANet code.zip.

Coal is an important energy source worldwide. Objectively and accurately predicting coal consumption is conducive to healthy coal industry development, because such predictions can provide references and warnings that are useful in formulating energy strategies and implementing environmental policies. Population size and area economic development are the main factors that affect coal consumption. Considering the above influences, this paper first establishes a differential equation and proposes a novel multivariable Verhulst grey model (MVGM(1,N)) based on grey information differences. MVGM(1,N) extends classical model from single-variable to multivariate and diminishes the characteristics of Verhulst’s reliance on saturated S-shaped and single-peak data, making classical model more applicable to real situations. To prove the effectiveness of MVGM(1,N) simulation experiments are carried out in areas with high coal consumption. The result of this proposed model is more precise than that of NLARX, ARIMA and five classical grey models Finally, this novel multivariable model predicates coal consumption of Inner Mongolia and Gansu Provinces in China, the results show that MVGM(1,N) is preferable to other models, indicating that this model can effectively predict coal consumption.

A traditional neural network cannot realize the invariance of image rotation and distortion well, so an attacker can fool the neural network by adding tiny disturbances to an image. If traffic signs are attacked, automatic driving will probably be misguided, leading to disastrous consequences. Inspired by the principle of human vision, this paper proposes a defense method based on an attentional mechanism for traffic sign adversarial samples. In this method, the affine coordinate parameters of the target objects in the images are extracted by a CNN, and then the target objects are redrawn by the coordinate mapping model. In this process, the key areas in the image are extracted by the attention mechanism, and the pixels are filtered by interpolation. Our model simulates the daily behavior of human beings, making it more intelligent in the defense against the adversarial samples. Experiments show that our method has a strong defense ability for traffic sign adversarial samples generated by various attack methods. Compared with other defense methods, our method is more universal and has a strong defense ability against a variety of attacks. Moreover, our model is portable and can be easily implanted into neural networks in the form of defense plug-ins.

To solve the problems of premature convergence and easily falling into local optimum, a whale optimization algorithm based on dynamic pinhole imaging and adaptive strategy is proposed in this paper. In the exploitation phase, the dynamic pinhole imaging strategy allows the whale population to approach the optimal solution faster, thereby accelerating the convergence speed of the algorithm. In the exploration phase, adaptive inertial weights based on dynamic boundaries and dimensions can enrich the diversity of the population and balance the algorithm’s exploitation and exploration capabilities. The local mutation mechanism can adjust the search range of the algorithm dynamically. The improved algorithm has been extensively tested in 20 well-known benchmark functions and four complex constrained engineering optimization problems, and compared with the ones of other improved algorithms presented in literatures. The test results show that the improved algorithm has faster convergence speed and higher convergence accuracy and can effectively jump out of the local optimum.

Person reidentification (reID) by convolutional neural network (CNN)-based networks has achieved favorable performance in recent years. However, most of existing CNN-based methods do not take full advantage of spatial–temporal context modeling. In fact, the global spatial–temporal context can greatly clarify local distractions to enhance the target feature representation. To comprehensively leverage the spatial–temporal context information, in this work, we present a novel block, interaction–aggregation-update (IAU), for high-performance person reID. First, the spatial–temporal IAU (STIAU) module is introduced. STIAU jointly incorporates two types of contextual interactions into a CNN framework for target feature learning. Here, the spatial interactions learn to compute the contextual dependencies between different body parts of a single frame, while the temporal interactions are used to capture the contextual dependencies between the same body parts across all frames. Furthermore, a channel IAU (CIAU) module is designed to model the semantic contextual interactions between channel features to enhance the feature representation, especially for small-scale visual cues and body parts. Therefore, the IAU block enables the feature to incorporate the globally spatial, temporal, and channel context. It is lightweight, end-to-end trainable, and can be easily plugged into existing CNNs to form IAUnet. The experiments show that IAUnet performs favorably against state of the art on both image and video reID tasks and achieves compelling results on a general object categorization task. The source code is available at https://github.com/blue-blue272/ImgReID-IAnet .

Signed digraphs have been developed in multiagent systems to explain cooperative and competitive interactions among agents. This article explores the collective behavior for a group of generic linear agents interacting by the random interaction manner on signed digraphs in the presence of multiple leaders. In the setting of random interactions, the information interaction between neighboring agents may fail at each time step, which is described by a Bernoulli random variable. The global almost sure convergence of the system is solved by using the convergence analysis method of products of infinite substochastic matrices. Moreover, an application about opinion dynamics of social networks is given. In the end, the theoretical research results are verified through simulation experiments.

We present a virtual image refocusing method over an extended depth of field (DOF) enabled by cascaded neural networks and a double-helix point-spread function (DH-PSF). This network model, referred to as W-Net, is composed of two cascaded generator and discriminator network pairs. The first generator network learns to virtually refocus an input image onto a user-defined plane, while the second generator learns to perform a cross-modality image transformation, improving the lateral resolution of the output image. Using this W-Net model with DH-PSF engineering, we extend the DOF of a fluorescence microscope by ~20-fold. This approach can be applied to develop deep learning-enabled image reconstruction methods for localization microscopy techniques that utilize engineered PSFs to improve their imaging performance, including spatial resolution and volumetric imaging throughput., 7 Pages, 3 Figures, 1 Table

This paper presents a compressed sensing (CS) based encryption scheme for wireless neural recording. An ultrahigh efficiency was achieved by leveraging CS for simultaneous data compression and encryption. CS enables sub-Nyquist sampling of neural signals by taking advantage of their intrinsic sparsity, while the CS process simultaneously encrypts the data with the sampling matrix being the cryptographic key. To share the key over an insecure wireless channel, we implemented an elliptic-curve cryptography (ECC) based key exchanging protocol. Local key shuffle and updating were adopted to eliminate the risks of potential information leakage. CS was executed in an application-specific integrated circuits (ASIC) fabricated in 180nm CMOS technology. Mixed-signal circuits were designed to optimize the power efficiency of the matrix-vector multiplication (MVM) of the CS operation. The ECC was implemented in a low-power Cortex-M0 based microcontroller (MCU). To be protected from timing attacks, the implementation avoided possible data-dependent branches. A wireless neural recorder prototype has been developed to demonstrate the proposed scheme. The prototype achieved an $8\times$ data rate reduction and a $35\times$ power saving compared with conventional implementation. The overall power consumption of ASIC and MCU was $442\mu \text{W}$ during the encrypted wireless transmission. The average correlated coefficient between the reconstructed signals and the uncompressed signals was 0.973, while the ciphertext-only attacks (CoA) achieved no better than 0.054 over 200,000 attacks. This work demonstrates a promising data compression and encryption scheme that can be used in a wide range of low-power signal recording systems with security requirements.

Background There is a large group of deaf-mutes in the world, and sign language is their major communication tool. Therefore, it’s necessary for the deaf-mutes to communicate with hearing-speech people and the hearing-speech people also have needed to understand sign language, which produces a great demand for sign language teaching. Even though there have already been a large number of books for sign language, it is low efficient to learn sign language with books, even teaching videos. To solve this problem, we develop a smartphone-based interactive Chinese sign language teaching systemfor sign language learning. Methods The system provides a learner with some kinds of learning modes and captures the learner’s actions from its front camera of the smartphone. Right now the system provides a vocabulary set with 1000 frequently used words, and the learner can evaluate his/her sign action by subjective or objective comparison. In the mode of word recognition, the users can play any word within the vocabulary and the system will turn the top three retrieved candidates, so it can remind the learners what the sign is. Results This system provides interactive learning for a user to learn sign language high efficiently. The systemadopts an algorithm based on point cloud recognition to evaluate a user’s sign and costs about 700ms inference time for each sample, which meets the real-time requirements. Conclusion This interactive learning system decreases the communication barriers between the deaf-mutes and hearing-speechers.

Digital holography is one of the most widely used label-free microscopy techniques in biomedical imaging. Recovery of the missing phase information of a hologram is an important step in holographic image reconstruction. Here we demonstrate a convolutional recurrent neural network (RNN) based phase recovery approach that uses multiple holograms, captured at different sample-to-sensor distances to rapidly reconstruct the phase and amplitude information of a sample, while also performing autofocusing through the same network. We demonstrated the success of this deep learning-enabled holography method by imaging microscopic features of human tissue samples and Papanicolaou (Pap) smears. These results constitute the first demonstration of the use of recurrent neural networks for holographic imaging and phase recovery, and compared with existing methods, the presented approach improves the reconstructed image quality, while also increasing the depth-of-field and inference speed., Comment: 18 Pages, 7 Figures, 1 Table

In this paper, we address the semantic segmentation task with a new context aggregation scheme named object context, which focuses on enhancing the role of object information. Motivated by the fact that the category of each pixel is inherited from the object it belongs to, we define the object context for each pixel as the set of pixels that belong to the same category as the given pixel in the image. We use a binary relation matrix to represent the relationship between all pixels, where the value one indicates the two selected pixels belong to the same category and zero otherwise. We propose to use a dense relation matrix to serve as a surrogate for the binary relation matrix. The dense relation matrix is capable to emphasize the contribution of object information as the relation scores tend to be larger on the object pixels than the other pixels. Considering that the dense relation matrix estimation requires quadratic computation overhead and memory consumption w.r.t. the input size, we propose an efficient interlaced sparse self-attention scheme to model the dense relations between any two of all pixels via the combination of two sparse relation matrices. To capture richer context information, we further combine our interlaced sparse self-attention scheme with the conventional multi-scale context schemes including pyramid pooling (Zhao et al. 2017) and atrous spatial pyramid pooling (Chen et al. 2018). We empirically show the advantages of our approach with competitive performances on five challenging benchmarks including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff.

Terahertz communication technology can provide abundant frequency resources, strong confidentiality, antijamming capability, communication tracking capability and the ability to achieve highspeed data transmissions and can serve as an important technical method for high-speed communications in the future. Among these terahertz communication technologies, terahertz direct modulation technology is a key means to achieve low system complexity and power consumption. In this paper, a review and outlook of terahertz direct modulation technology are proposed from the aspects of high-electron-mobility-transistor-based terahertz direct modulation, parallelswitch terahertz direct modulation, diode-based terahertz direct modulation, quantum cascade laser-based terahertz direct modulation and new-material-based terahertz direct modulation. We hope through this paper that more readers can gain knowledge about the development and challenges of terahertz direct modulation technology for high-speed communication systems, thus promoting the development of high-speed terahertz communication technology based on direct modulation.

State-of-the-art classification models are usually considered as black boxes since their decision processes are implicit to humans. On the contrary, human experts classify objects according to a set of explicit hierarchical criteria. For example, “ tabby is a domestic cat with stripes, dots, or lines”, where tabby is defined by combining its superordinate category ( domestic cat ) and some certain attributes (e.g., has stripes). Inspired by this mechanism, we propose an interpretable Hierarchical Criteria Network (HCN) by additionally learning such criteria. To achieve this goal, images and semantic entities (e.g., taxonomies and attributes) are embedded into a common space, where each category can be represented by the linear combination of its superordinate category and a set of learned discriminative attributes. Specifically, a two-stream convolutional neural network (CNN) is elaborately devised, which embeds images and taxonomies with the two streams respectively. The model is trained by minimizing the prediction error of hierarchy labels on both streams. Extensive experiments on two widely studied datasets (CIFAR-100 and ILSVRC) demonstrate that HCN can learn meaningful attributes as well as reasonable and interpretable classification criteria. Therefore, the proposed method enables further human feedback for model correction as an additional benefit.

The remaining useful life (RUL) prediction of bearings plays a crucial role in ensuring the safe operation of machinery and reducing costly maintenance. Sufficient degradation information contributes to the accuracy of RUL prediction, but the effective retention of degradation information is still a difficult task, especially for small samples. Thus, a novel RUL prediction method was proposed using data augmentation (DA) and a deep bidirectional gate recurrent unit (DBGRU). Initially, a feature selection method without consideration of the operating time that exploited the mutual information of different units, namely, the main factor correlation index (MFCI), was proposed to construct suitable feature input. In addition, a higher volume of training samples was generated with an effective DA technique, Mixup, and a bidirectional gate recurrent unit (BGRU) layer was utilized to further explore the subtle degradation information with the mixed data and the raw data, which also improved the robustness and generalization of the model. Ultimately, the advantage of the proposed method was demonstrated by a comparison with several state-of-the-art prediction models for the same circumstance. The test results showed that the proposed method was promising and that it had good prediction accuracy and robustness for different working conditions.

Approximate computing is tentatively applied in some digital signal processing applications which have an inherent tolerance for erroneous computing results. The approximate arithmetic blocks are utilized in them to improve the electrical performance of these circuits. Multiplier is one of the fundamental units in computer arithmetic blocks. Moreover, the 4–2 compressors are widely employed in the parallel multipliers to accelerate the compression process of partial products. In this brief, three novel approximate 4–2 compressors are proposed and utilized in 8-bit multipliers. Meanwhile, an error-correcting module (ECM) is presented to promote the error performance of approximate multiplier with the proposed 4–2 compressors. In this brief, the number of the approximate 4–2 compressor’s outputs is innovatively reduced to one, which brings further improvements in the energy-efficiency. Compared with the exact 4–2 compressors, the simulation results indicate that the proposed approximate compressors UCAC1, UCAC2, UCAC3 achieve 24.76%, 51.43%, and 66.67% reduction in delay, 71.76%, 83.06%, and 93.28% reduction in power and 54.02%, 79.32%, and 93.10% reduction in area, respectively. And the utilization of these proposed compressors in 8-bit multipliers brings 49.29% reduction of power consumption on average.

Face presentation attack detection (PAD) is essential for securing the widely used face recognition systems. Most of the existing PAD methods do not generalize well to unseen scenarios because labeled training data of the new domain is usually not available. In light of this, we propose an unsupervised domain adaptation with disentangled representation (DR-UDA) approach to improve the generalization capability of PAD into new scenarios. DR-UDA consists of three modules, i.e., ML-Net, UDA-Net and DR-Net. ML-Net aims to learn a discriminative feature representation using the labeled source domain face images via metric learning. UDA-Net performs unsupervised adversarial domain adaptation in order to optimize the source domain and target domain encoders jointly, and obtain a common feature space shared by both domains. As a result, the source domain PAD model can be effectively transferred to the unlabeled target domain for PAD. DR-Net further disentangles the features irrelevant to specific domains by reconstructing the source and target domain face images from the common feature space. Therefore, DR-UDA can learn a disentangled representation space which is generative for face images in both domains and discriminative for live vs. spoof classification. The proposed approach shows promising generalization capability in several public-domain face PAD databases.

Location is an important distinguishing information for instance segmentation. In this paper, we propose a novel model, called Location Sensitive Network (LSNet), for human instance segmentation. LSNet integrates instance-specific location information into one-stage segmentation framework. Specifically, in the segmentation branch, Pose Attention Module (PAM) encodes the location information into the attention regions through coordinates encoding. Based on the location information provided by PAM, the segmentation branch is able to effectively distinguish instances in feature-level. Moreover, we propose a combination operation named Keypoints Sensitive Combination (KSCom) to utilize the location information from multiple sampling points. These sampling points construct the points representation for instances via human keypoints and random points. Human keypoints provide the spatial locations and semantic information of the instances, and random points expand the receptive fields. Based on the points representation for each instance, KSCom effectively reduces the mis-classified pixels. Our method is validated by the experiments on public datasets. LSNet-5 achieves 56.2 mAP at 18.5 FPS on COCOPersons. Besides, the proposed method is significantly superior to its peers in the case of severe occlusion.

Structural displacement is an important quantity to assess the health of civil infrastructure. Vision‐based approaches using unmanned aerial vehicles (UAV) mounted with high‐resolution cam...

Large datasets that enable researchers to perform investigations with unprecedented rigor are growing increasingly common in neuroimaging. Due to the simultaneous increasing popularity of open science, these state-of-the-art datasets are more accessible than ever to researchers around the world. While analysis of these samples has pushed the field forward, they pose a new set of challenges that might cause difficulties for novice users. Here we offer practical tips for working with large datasets from the end-user’s perspective. We cover all aspects of the data lifecycle: from what to consider when downloading and storing the data to tips on how to become acquainted with a dataset one did not collect and what to share when communicating results. This manuscript serves as a practical guide one can use when working with large neuroimaging datasets, thus dissolving barriers to scientific discovery. Horien and colleagues provide a roadmap to working with large, publically available imaging datasets.

In order to gain substantial circuit performance improvement in terms of power, speed and area cost, approximate computing is ideally applied in many error-resilient DSP and multimedia applications by relaxing the requirements of exact computation. Fixed-width multipliers are intensively used in those applications, which makes them a good demonstration for approximation through dynamic range reduction. In this paper, a probabilistic prediction based fixed-width Booth multiplier has been proposed. The error compensation function is derived from the probabilistic analysis on the partial products obtained from Booth encoding. By slightly modifying the partial product array with dedicated partitioning, a simple compensation circuit is thus formulated. Compared with the previous works, the proposed design exhibits the best performance-accuracy tradeoff. It achieves at least 27% reduction on the area-energy-delay-error product compared with the other multipliers among various operand lengths. Additionally, the evaluation on 2-Dimensional discrete cosine transform (DCT) circuits indicates that our proposed fixed-width Booth multiplier is applicable for lossy applications with a considerable reduction on hardware expenses and power dissipation while maintaining decent output quality.

Achieving a comprehensive understanding of brain function requires multiple imaging modalities with complementary strengths. We present an approach for concurrent widefield optical and functional magnetic resonance imaging. By merging these modalities, we can simultaneously acquire whole-brain blood-oxygen-level-dependent (BOLD) and whole-cortex calcium-sensitive fluorescent measures of brain activity. In a transgenic murine model, we show that calcium predicts the BOLD signal, using a model that optimizes a gamma-variant transfer function. We find consistent predictions across the cortex, which are best at low frequency (0.009–0.08 Hz). Furthermore, we show that the relationship between modality connectivity strengths varies by region. Our approach links cell-type-specific optical measurements of activity to the most widely used method for assessing human brain function. Simultaneous widefield calcium imaging and fMRI provide insight into neural activity at multiple scales and can be used to decipher the cellular origin of BOLD activity.

With more and more color images are spread and shared in the untrusted cloud server remotely over the open Internet, some private or confidential information may be contained, it is necessary to develop cryptosystem for these images to free from serious threats. This paper investigates a secure and robust cryptosystem for color image based on discrete trinion Fourier transform and random-multiresolution singular value decomposition. Firstly, a color image is represented into a trinion matrix by precoding and followed by block-wise discrete trinion Fourier transform. Then a mapping from trinion number domain to real number domain is implemented to facilitate random-multiresolution singular value decomposition. To further enhance the randomness of the transformed data, the scrambling based on chaotic sequence is applied to obtain the cipherimage. Experimental results have demonstrated that the proposed color image encryption algorithm is feasible. The adoption of logistic map ensures that the proposed cryptosystem attains high level security. Furthermore, the quality of the retrieved image is better than that of the algorithm based on single color channel processing separately and quaternion matrix based algorithms when it is decrypted correctly or under Gaussian noise and shear attacks. Moreover, employing a trinion matrix to represent a color image is more compact than using quaternion matrix. It would have potential applications in digital image protection.

Chimeric antigen receptor (CAR) therapy has achieved remarkable clinical efficacy against hematological cancers and has been approved by FDA for treatment of B-cell tumors. However, the complex manufacturing process and limited success in solid tumors hamper its widespread applications, thus prompting the development of new strategies for overcoming the abovementioned hurdles. In the last decade, nanotechnology has provided sustainable strategies for improving cancer immunotherapy through vaccine development and delivery of immunomodulatory drugs. Nanotechnology can boost CAR-T therapy and may overcome the existing challenges by emerging as a carrier for CAR-T therapy or in combination with CAR-T, it may inhibit solid tumors more effectively than conventional approaches. The revealing of cellular mechanisms, barriers and potential strategies that could be used to manipulate and/or modify cells would enable unprecedented advances in nanotechnology for biologics delivery. This review outlines the journey and barriers of nanoparticles (NPs) across the cell. Subsequently, the approaches to tackle the barriers and strategies to modulate NPs as a carrier for CAR-T therapy are discussed. Finally, the role of NPs in CAR-T therapy and the potential challenges are summarized. This review aims to provide the readers with a detailed overview of NP-based CAR-T therapy research and distil this information into an accessible form conducive to design desired CAR-T therapy using NP approach. STATEMENT OF SIGNIFICANCE: Chimeric antigen receptor (CAR) T-cell therapy is the most vibrant field in immuno-oncology today, with enormous benefits to patients with B-cell malignancies. However, a rapid and straightforward procedure for CAR-T generation is an exigent need to broaden its therapeutic avenue. Nanotechnology has emerged as a novel alternative approach for CAR-T generation. To the best of our knowledge, this is the first in-depth review that briefly highlights the various aspects of nanotechnology in CAR-T therapy, including the strategies to brand NPs as an effective carrier for CAR cargo, its potential advantages, challenges, and future roadmap. It provides readers with a detailed overview of NP-based CAR-T therapy research, and researchers would be able to distill this information into an accessible form conducive to design the desired CAR therapy using the nanotechnology approach.

Learning the conjunction of multiple visual concepts shows practical significance in various real world applications (e.g. multi-attribute image retrieval and visual relationship detection). In this paper, we propose Concept Conjunction Recurrent Neural Network (C2RNN) to tackle this problem. With our model, visual concepts involved in a conjunction are mapped into the hidden units and combined in a recurrent way to generate the representation of the concept conjunction, which is then used to compute a concept conjunction classifier as the output. We also present an order invariant version of the proposed method based on attention mechanism to learn the tasks without pre-defined concept order. To tackle concept conjunction learning from multiple semantic domains, we introduce a multiplicative framework to learn the joint representation. Experimental results on multi-attribute image retrieval and visual relationship detection show that our method achieves significantly better performance than other related methods on various datasets.

In this work, we study the problem of deep representation learning on a large face dataset with long-tail distribution. Training convolutional neural networks on such dataset with conventional strategy suffers from imbalance problem which results in biased classification boundary, and the few-shot classes lying in tail parts further make the model prone to overfitting. Aiming to learn more discriminative CNN model from long-tail data, we propose a novel aggregate-and-disperse training schema. Firstly, our proposed method aggregates similar classes in tail part to avoid imbalance problem. Based on the aggregated super classes and those original head classes, a model is pre-trained to capture accurate discrimination in head classes as well as coarse discrinimation in tail classes. Secondly, we selectively disperses those aggregated super classes to learn precise inter-class variations and refine the representation for better generalization. We perform extensive experiments on MS-Celeb-1M, BLUFR and MegaFace. Compared with baselines and existing methods, our method achieves better performance of face recognition, demonstrating its effectiveness of handling long-tail distribution.

Devices such as keyboards and touchscreens allow humans to communicate with machines. Neural interfaces, which can provide a direct, electrical bridge between analogue nervous systems and digital man-made systems, could provide a more efficient route to future information exchange. Here we review the development of electronic neural interfaces. The interfaces typically consist of three modules — a tissue interface, a sensing interface, and a neural signal processing unit — and based on technical milestones in the development of the electronic sensing interface, we group and analyse the interfaces in four generations: the patch clamp technique, multi-channel neural interfaces, implantable/wearable neural interfaces and integrated neural interfaces. We also consider key circuit and system challenges in the design of neural interfaces and explore the opportunities that arise with the latest technology This Review Article examines the development of neural interfaces, which can provide a direct, electrical bridge between analogue human nervous systems and digital man-made devices, considering challenges and opportunities created with such technology.

Functionality is a fundamental attribute of an object which indicates the capability to be used to perform specific actions. It is critical to empower robots the functionality knowledge in discovering appropriate objects for a task e.g. cut cake using knife. Existing research works have focused on understanding object functionality through human-object-interaction from extensively annotated image or video data and are hard to scale up. In this paper, we (1) mine object-functionality knowledge through pattern-based and model-based methods from text, (2) introduce a novel task on physical object-functionality prediction, which consumes an image and an action query to predict whether the object in the image can perform the action, and (3) propose a method to leverage the mined functionality knowledge for the new task. Our experimental results show the effectiveness of our methods.

Due to the highly complex semantic information of images, even with the same query image, the expected content-based image retrieval results could be very different and personalized in different scenarios. However, most existing hashing methods only preserve one single type of semantic similarity, making them incapable of addressing such realistic retrieval tasks. To deal with this problem, we propose a unified hashing framework to encode multiple types of information into the binary codes by exploiting convolutional networks (CNNs). Specifically, we assume that typical retrieval tasks are generally defined in two aspects, i.e. high-level semantics (e.g. object categories) and visual attributes (e.g. object shape and color). To this end, our Dual Purpose Hashing model is trained to jointly preserve two kinds of similarities characterizing the two aspects respectively. Moreover, since images with both category and attribute labels are scarce, our model is carefully designed to leverage the abundant partially labelled data as training inputs to alleviate the risk of overfitting. With such a framework, the binary codes of new-coming images can be readily obtained by quantizing the outputs of a specific CNN layer, and different retrieval tasks can be achieved by using the binary codes in different ways. Experiments on two large-scale datasets show that our method achieves comparable or even better performance than those state-of-the-art methods specifically designed for each individual retrieval task while being more compact than the compared methods.

Solving visual question answering (VQA) task requires recognizing many diverse visual concepts as the answer. These visual concepts contain rich structural semantic meanings, e.g., some concepts in VQA are highly related (e.g., red & blue), some of them are less relevant (e.g., red & standing). It is very natural for humans to efficiently learn concepts by utilizing their semantic meanings to concentrate on distinguishing relevant concepts and eliminate the disturbance of irrelevant concepts. However, previous works usually use a simple MLP to output visual concept as the answer in a flat label space that treats all labels equally, causing limitations in representing and using the semantic meanings of labels. To address this issue, we propose a novel visual recognition module named Dynamic Concept Recognizer (DCR), which is easy to be plugged in an attention-based VQA model, to utilize the semantics of the labels in answer prediction. Concretely, we introduce two key features in DCR: 1) a novel structural label space to depict the difference of semantics between concepts, where the labels in new label space are assigned to different groups according to their meanings. This type of semantic information helps decompose the visual recognizer in VQA into multiple specialized sub-recognizers to improve the capacity and efficiency of the recognizer. 2) A feature attention mechanism to capture the similarity between relevant groups of concepts, e.g., human-related group “chef, waiter” is more related to “swimming, running, etc.” than scene related group “sunny, rainy, etc.”. This type of semantic information helps sub-recognizers for relevant groups to adaptively share part of modules and to share the knowledge between relevant sub-recognizers to facilitate the learning procedure. Extensive experiments on several datasets have shown that the proposed structural label space and DCR module can efficiently learn the visual concept recognition and benefit the performance of the VQA model.

Advancement in single-cell RNA sequencing leads to exponential accumulation of single-cell expression data. However, there is still lack of tools that could integrate these unlimited accumulation of single-cell expression data. Here, we presented a universal approach iSEEEK for integrating super large-scale single-cell expression via exploring expression rankings of top-expressing genes. We developed iSEEEK with 13.7 million single-cells. We demonstrated the efficiency of iSEEEK with canonical single-cell downstream tasks on five heterogenous datasets encompassing human and mouse samples. iSEEEK achieved good clustering performance benchmarked against well-annotated cell labels. In addition, iSEEEK could transfer its knowledge learned from large-scale expression data on new dataset that was not involved in its development. iSEEEK enables identification of gene-gene interaction networks that are characteristic of specific cell types. Our study presents a simple and yet effective method to integrate super large-scale single-cell transcriptomes and would facilitate translational single-cell research from bench to bedside.

We introduce a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD). Traditional methods for ASD usually operate on each candidate's pre-cropped face track separately and do not sufficiently consider the relationships among the candidates. This potentially limits performance, especially in challenging scenarios with low-resolution faces, multiple candidates, etc. Our solution is a novel, unified framework that focuses on jointly modeling multiple types of contextual information: spatial context to indicate the position and scale of each candidate's face, relational context to capture the visual relationships among the candidates and contrast audio-visual affinities with each other, and temporal context to aggregate long-term information and smooth out local uncertainties. Based on such information, our model optimizes all candidates in a unified process for robust and reliable ASD. A thorough ablation study is performed on several challenging ASD benchmarks under different settings. In particular, our method outperforms the state-of-the-art by a large margin of about 15% mean Average Precision (mAP) absolute on two challenging subsets: one with three candidate speakers, and the other with faces smaller than 64 pixels. Together, our UniCon achieves 92.0% mAP on the AVA-ActiveSpeaker validation set, surpassing 90% for the first time on this challenging dataset at the time of submission. Project website: https://unicon-asd.github.io/., Comment: 10 pages, 6 figures; to appear at ACM Multimedia 2021

In this paper, we propose a method to realize complex amplitude modulation with a 4-level phase plate and an amplitude-modulated SLM. We implement this complex amplitude modulator system in classical 4f system and use this system to realize highly precise convolution to verify its feasibility. The optical convolution accuracy is higher than the phase-only modulator or the amplitude-only modulator method. We then incorporate this system with a one-layer CNN as its convolutional layer to accomplish a handwriting recognition inference, the image classification accuracy is also comparable to all-electrical result with the same framework.

Aiming at the problems of poor welding quality and low degree of automatic welding on the engineering site, a welding process parameter control method based on machine vision and nonlinear regression technology is proposed. Firstly, a vision unit and a peripheral sensor unit are designed to obtain the information of each influencing factor of the welding process parameters. Secondly, a clustering algorithm is used to improve the extraction accuracy of feature point coordinates of weld images. Thirdly, a nonlinear regression fitting method is proposed to determine the mathematical relationship between welding quality at different welding positions and corresponding process parameters. Experimental results show that the control system is easy to operate, and the flexible control of welding process parameters in the whole process is realized. The weld cumulative height and width deviations are less than 0.5 and 0.3 mm, respectively. The welding surface is stable and meets welding requirements. Therefore, this method is of great practical significance in engineering field welding.

After an accurate polyp detection, precise polyp dissection in a safe zone by the robot-assisted polypectomy is required. Virtual Fixtures (VF) provide surgeons with tactile force feedback and help operators complete high-precision and safe works successfully. This paper presents a safe and high-precise polyp dissection method combined guidance virtual fixtures and forbidden region virtual fixtures. The guidance virtual fixtures are created using a 3D curve interpolated by defining the burning points during polypectomy, and can be changed at any time according to the surgeon’s needs. The prohibited areas are generated based on 3D point clouds to prevent surgeons from harming the healthy tissues. The proposed method is verified by physical robot experiments, and the results show that it can improves both precision and safety.

Spontaneous and sensory-evoked activity propagates across varying spatial scales in the mammalian cortex, but technical challenges have limited conceptual links between the function of local neuronal circuits and brain-wide network dynamics. We present a method for simultaneous cellular-resolution two-photon calcium imaging of a local microcircuit and mesoscopic widefield calcium imaging of the entire cortical mantle in awake mice. Our multi-scale approach employs an orthogonal axis design where the mesoscopic objective is oriented above the brain and the two-photon objective is oriented horizontally, with imaging performed through a microprism. We also introduce a viral method for robust and widespread gene delivery in the mouse brain. These approaches allow us to identify the behavioral state-dependent functional connectivity of pyramidal neurons and vasoactive intestinal peptide (VIP)-expressing interneurons with long-range cortical networks. Our novel imaging system provides a powerful strategy for investigating cortical architecture across a wide range of spatial scales., Reporting summary Further information on research design is available in the Life Sciences Reporting Summary linked to this article.

Objective The profession of surgery is entering a new era of “big data,” where analyses of longitudinal trainee assessment data will be used to inform ongoing efforts to improve surgical education. Given the high-stakes implications of these types of analyses, researchers must define the conditions under which estimates derived from these large datasets remain valid. With this study, we determine the number of assessments of residents’ performances needed to reliably assess the difficulty of “Core” surgical procedures. Design Using the SIMPL smartphone application from the Procedural Learning and Safety Collaborative, 402 attending surgeons directly observed and provided workplace-based assessments for 488 categorical residents after 5259 performances of 87 Core surgical procedures performed at 14 institutions. We used these faculty ratings to construct a linear mixed model with resident performance as the outcome variable and multiple predictors including, most significantly, the operative procedure as a random effect. We interpreted the variance in performance ratings attributable to the procedure, after controlling for other variables, as the “difficulty” of performing the procedure. We conducted a generalizability analysis and decision study to estimate the number of SIMPL performance ratings needed to reliably estimate the difficulty of a typical Core procedure. Results Twenty-four faculty ratings of resident operative performance were necessary to reliably estimate the difficulty of a typical Core surgical procedure (mean dependability coefficient 0.80, 95% confidence interval 0.73-0.87). Conclusions At least 24 operative performance ratings are required to reliably estimate the difficulty of a typical Core surgical procedure. Future research using performance ratings to establish procedure difficulty should include adequate numbers of ratings given the high-stakes implications of those results for curriculum design and policy.

Facial attribute editing aims to manipulate single or multiple attributes of a face image, i.e., to generate a new face with desired attributes while preserving other details. Recently, generative adversarial net (GAN) and encoder-decoder architecture are usually incorporated to handle this task with promising results. Based on the encoder-decoder architecture, facial attribute editing is achieved by decoding the latent representation of the given face conditioned on the desired attributes. Some existing methods attempt to establish an attribute-independent latent representation for further attribute editing. However, such attribute-independent constraint on the latent representation is excessive because it restricts the capacity of the latent representation and may result in information loss, leading to over-smooth and distorted generation. Instead of imposing constraints on the latent representation, in this work we apply an attribute classification constraint to the generated image to just guarantee the correct change of desired attributes, i.e., to "change what you want". Meanwhile, the reconstruction learning is introduced to preserve attribute-excluding details, in other words, to "only change what you want". Besides, the adversarial learning is employed for visually realistic editing. These three components cooperate with each other forming an effective framework for high quality facial attribute editing, referred as AttGAN. Furthermore, our method is also directly applicable for attribute intensity control and can be naturally extended for attribute style manipulation. Experiments on CelebA dataset show that our method outperforms the state-of-the-arts on realistic attribute editing with facial details well preserved., Submitted to IEEE Transactions on Image Processing, Code: https://github.com/LynnHo/AttGAN-Tensorflow

Visual sign language recognition is an interesting and challenging problem. To create a discriminative representation, a hierarchical Grassmann covariance matrix (HGCM) model is proposed for sign description. Furthermore, a multi-temporal belief propagation (MTBP) based segmentation approach is presented for continuous sequence spotting. Concretely speaking, a sign is represented by multiple covariance matrices, followed by evaluating and selecting their most significant singular vectors. These covariance matrices are transformed into a more compact and discriminative HGCM, which is formulated on the Grassmann manifold. Continuous sign sequences can be recognized frame by frame using the HGCM model, before being optimized by MTBP, which is a carefully designed graphic model. The proposed method is thoroughly evaluated on isolated and synthetic and real continuous sign datasets as well as on HDM05. Extensive experimental results convincingly show the effectiveness of our proposed framework.

In this contribution, the containment control problem for a team of agents with discrete-time second-order dynamics over dynamic cooperative networks is discussed, where data from the controller to the actuator may be lost randomly and it is described by a random variable obeying Bernoulli distribution. A random-based distributed controller is designed by using the information from neighbors. Based on the hybrid tools of graph theory and nonnegative matrix, it is shown that the implementation of containment control is related to the dynamic cooperative networks and the successful rate of information transmission. Finally, a simulation is carried out to demonstrate the result in this paper.

The explosive growth of digital images in video surveillance and social media has led to the significant need for efficient search of persons of interest in law enforcement and forensic applications. Despite tremendous progress in primary biometric traits (e.g., face and fingerprint) based person identification, a single biometric trait alone cannot meet the desired recognition accuracy in forensic scenarios. Tattoos, as one of the important soft biometric traits, have been found to be valuable for assisting in person identification. However, tattoo search in a large collection of unconstrained images remains a difficult problem, and existing tattoo search methods mainly focus on matching cropped tattoos, which is different from real application scenarios. To close the gap, we propose an efficient tattoo search approach that is able to learn tattoo detection and compact representation jointly in a single convolutional neural network (CNN) via multi-task learning. While the features in the backbone network are shared by both tattoo detection and compact representation learning, individual latent layers of each sub-network optimize the shared features toward the detection and feature learning tasks, respectively. We resolve the small batch size issue inside the joint tattoo detection and compact representation learning network via random image stitch and preceding feature buffering. We evaluate the proposed tattoo search system using multiple public-domain tattoo benchmarks, and a gallery set with about 300K distracter tattoo images compiled from these datasets and images from the Internet. In addition, we also introduce a tattoo sketch dataset containing 300 tattoos for sketch-based tattoo search. Experimental results show that the proposed approach has superior performance in tattoo detection and tattoo search at scale compared to several state-of-the-art tattoo retrieval algorithms., Comment: Technical Report (15 pages, 14 figures)

Zero-shot learning (ZSL) has enjoyed great popularity in recent years due to its ability to recognize novel objects, where semantic information is exploited to build up relations among different categories. Traditional ZSL approaches usually focus on learning more robust visual-semantic embeddings among seen classes and directly apply them to the unseen classes without considering whether they are suitable. It is well known that domain gap exists between seen and unseen classes. In order to tackle such problem, we propose a novel adaptive metric learning approach to measure the compatibility between visual samples and class semantics, where class similarities are utilized to adapt the visual-semantic embedding to the unseen classes. Extensive experiments on four benchmark ZSL datasets show the effectiveness of the proposed approach.

In this paper, a novel adaptive active contour model based on image data field for image segmentation with robust and flexible initializations is proposed. We firstly construct a new external energy term deduced from the image data field that drives the level set function to move in the opposite direction along the boundaries of object and an adaptive length regularization term based on the image local entropy. The designed external energy and length regularization term are then incorporated into a variationlevel set framework with an additional penalizing energy term. Due to the adaptive sign–changing property of the external energy and the adaptive length regularization term, the proposed model can tackle images with clutter background and noise, the level set function can be initialized as any bounded functions (e.g., constant function), which implies the proposed model is robust to initialization of contours. Experimental results on both synthetic and real images from different modalities confirm the effectiveness and competivive performance of the proposed method compared with other representative models.