Search

Mutli-camera vehicle tracking and re-identification (re-ID) have gradually gained attention due to their applications in the intelligent transportation system. However, these problems are fundamentally challenging. Specifically, for vehicle tracking, we observe that the results generated from single camera tracking algorithm usually recognize tracklets with same identity as different vehicles when the tracklets are occluded. Hence, we propose a Tracklet Re-connection technique to refine tracking results with pre-defined zone areas and GPS information. The proposed method can efficiently filter invalid tracklet pairs and reconnect the split tracklets into complete ones, which is important for the afterwards multi-target multi-camera tracking. As for re-ID, we also find that when a large-scale auxiliary dataset is used to assist the learning of main dataset for better model capability and generalization, there is a performance drop caused by data imbalance when the full auxiliary dataset is applied. To tackle this problem, we introduce Balanced Cross-Domain Learning to avoid the overemphasis on larger auxiliary dataset by a newly introduced training data sampler and loss function. The extensive experiments validate the empirical effectiveness of our proposed components.

We propose a dense tensor accelerator called VectorMesh, a scalable, memory-efficient architecture that can support a wide variety of DNN and computer vision workloads. Its building block is a tile execution unit (TEU), which includes dozens of processing elements (PEs) and SRAM buffers connected through a butterfly network. A mesh of FIFOs between the TEUs facilitates data exchange between tiles and promote local data to global visibility. Our design performs better according to the roofline model for CNN, GEMM, and spatial matching algorithms compared to state-of-the-art architectures. It can reduce global buffer and DRAM fetches by 2-22 times and up to 5 times, respectively.

Person re-identification (Re-ID) aims to match images of the same person across distinct camera views. In this paper, we propose the Space-Time Guided Association Learning (STGAL) for unsupervised Re-ID without ground truth identity nor image correspondence observed during training. By exploiting the spatial-temporal information presented in pedestrian data, our STGAL is able to identify positive and negative image pairs for learning Re-ID feature representations. Experiments on a variety of datasets confirm the effectiveness of our approach, which achieves promising performance when comparing to the state-of-the-art methods.

Deep neural networks have demonstrated unprecedented results on core AI and computer vision tasks. They are typically executed on general purpose GPUs with large form factors and high power consumption, unsuitable for mobile deployment. We present a VLSI architecture that is able to execute quantized, low precision convolution neural networks (CNNs). Compared to high precision, our approach significantly reduces power consumption from memory access and increases processing speed at limited area budget, making it particularly suitable for mobile applications. We propose a dataflow with high data reuse rate specially designed for quantized models. To fully utilize low precision data, we also design a microarchitecture for subword parallel computing of low bit-length data, an on-chip memory hierarchy and data realignment flow for power saving and avoiding buffer bank-conflicts, and finally a corresponding processing element (PE) array. The architecture is highly flexible to suit various CNNs and re-configurable for low bit-length quantized models. We have implemented the proposed VLSI architecture in the TSMC 90nm cell library. At a hardware cost of 180KB on-chip memory and 1,340k logic gate counts, the implementation result shows state-of-the-art hardware efficiency.

Although convolutional neural networks (CNNs) have revolutionized many computer vision tasks in recent years, spatial propagation algorithm still plays an important role in many applications such as image matting, segmentation, depth estimation, and colorization. However, most of these methods focus on qualitative performance and result in expensive computation, which is not conducive to be deployed on mobile or edge devices. In this paper, we take the spatial propagation network (SPN), convolutional spatial propagation network (CSPN), and guided convolution as our analysis objects. We evaluate these methods from the hardware accelerating aspect and use depth completion as an example. In addition to quality comparison, we propose a hardware feasibility metric that analyzes each method in terms of hardware resources. With this metric, developers can easily decide a proper algorithm on a specific hardware platform. We apply it to an example and get a consistent result, successfully building the connection between algorithm and hardware design.

This paper presents a super-resolution CNN de-signed for real-time hardware processing and the associated hardware architecture. Previous networks typically contain numerous layers, various kernel sizes and deconvolution layers, making it hard for hardware implementation. In this paper, we present a CNN only consisting of 3×3 convolution, replacing the deconvolution by pixel shuffling. Such regularity of kernel size enables us to employ Winograd convolution to implement the whole network. The proposed architecture achieves output resolution of 1920×1080 (FHD) at 60 fps while working at a clock frequency of 200 MHz. It also outperforms other 12K-parameter networks in image quality.

Filter pruning is an efficient way to structurally remove the redundant parameters in convolutional neural network, where at the same time reduces the computation, memory storage and transfer cost. Recent state-of-the-art methods globally estimate the importance of each filter based on its impact to the loss and iteratively remove those with smaller values until the pruned network meets some resource constraints, such as the commonly used number (or ratio) of filter left. However, when there is a more practical constraint like the total number of FLOPs, they ignore its relation to the estimation of filter importance. We propose a novel method called Constraint-Aware Importance Estimation (CAIE) that integrates information of the impact on the given resource into the original importance estimation only based on loss when pruning each filter. Moreover, our CAIE can be generalized to the pruning problem under multiple resource constraints simultaneously. Extensive experiments show that under the same multiple resource constraints, the model pruned with our CAIE method can not only accurately meet the constraints but also achieve the optimal performance results when comparing to existing state-of-the-art methods.

Vehicle re-identification (re-ID) matches images of the same vehicle across different cameras. It is fundamentally challenging because the dramatically different appearance caused by different viewpoints would make the framework fail to match two vehicles of the same identity. Most existing works solved the problem by extracting viewpoint-aware feature via spatial attention mechanism, which, yet, usually suffers from noisy generated attention map or otherwise requires expensive keypoint labels to improve the quality. In this work, we propose Viewpoint-aware Channel-wise Attention Mechanism (VCAM) by observing the attention mechanism from a different aspect. Our VCAM enables the feature learning framework channel-wisely reweighing the importance of each feature maps according to the "viewpoint" of input vehicle. Extensive experiments validate the effectiveness of the proposed method and show that we perform favorably against state-of-the-arts methods on the public VeRi-776 dataset and obtain promising results on the 2020 AI City Challenge. We also conduct other experiments to demonstrate the interpretability of how our VCAM practically assists the learning framework., CVPR Workshop 2020

Recently, deep neural networks have led to tremendous advances in image super-resolution. As a well-known one-to-many inverse problem, the deep learning based methods tackle this issue via large receptive field. By that, the deep network could infer each output pixel from sufficient context information. However, most existing studies use larger kernel size or design a very deep network model to attain sufficient receptive field. The computational cost dramatically increments along with the training difficulty. Concerning this problem, the goal of this paper is to design an effective and trainable convolutional neural network. We proposed a multi-scale dense network (MSDN) which is composed of deep concatenation and basic blocks, namely multi-scale dense block (MSDB). The proposed MSDB use different dilated convolutions to gather multi-scale information; meanwhile concatenating the different dilated convolution results magnify the receptive field of a single layer. To facilitate the training difficulty, there are the dense skip connections in the proposed MSDB. Moreover, the deep concatenation and global skip connection are also adopted for improving training furthermore. Consequently, we achieve a large receptive field network without deeper structure. The experiments indicate that the quality of the proposed MSDN yields the state-of-the-art result.

Deep neural networks (DNNs) have demonstrated impressive performance in many edge computer vision tasks, causing the increasing demand for DNN accelerator on mobile and internet of things (IoT) devices. However, the massive power consumption and storage requirement make the hardware design challenging. In this paper, we introduce a DNN accelerator based on a model compression technique vector quantization (VQ), which can reduce the network model size and computation cost simultaneously. Moreover, a specialized processing element (PE) is designed with various SRAM bank configurations as well as dataflows such that it can support different codebook/kernel sizes, and keep high utilization under small input or output channel numbers. Compared to the state-of-the-art, the proposed accelerator architecture achieves 4.2 times reduction in memory access and 2.05 times throughput per cycle for batch-one inference.

In recent years, deep convolutional neural networks (CNNs) achieve ground-breaking success in many computer vision research fields. Due to the large model size and tremendous computation of CNNs, they cannot be efficiently executed in small devices like mobile phones. Although several hardware accelerator architectures have been developed, most of them can only efficient address one of the two major layers in CNN, convolutional (CONV) and fully connected (FC) layers. In this paper, based on algorithm-architecture-co-exploration, our architecture targets at executing both layers with high efficiency. Vector quantization technique is first selected to compress the parameters, reduce the computation, and unify the behaviors of both CONV and FC layers. To fully exploit the gain of vector quantization, we then propose an accelerator architecture for quantized CNN. Different DRAM access schemes are employed to reduce DRAM access. We also design a high-throughput processing element architecture to accelerate quantized layers. Compare to previous accelerators for CNN, the proposed architecture achieves 1.2–5x less DRAM access and 1.5–5x higher throughput for both CONV and FC layers.

To accelerate Convolutional Neural Networks (CNN) operations on resource-limited mobile graphics processing units (GPUs), taking advantage of the common characteristics between texture filtering and convolutional layer, we propose a configurable texture unit called tensor and texture unit (TTU) to offload the computation from shader cores. With adding a new datapath for loading weight parameters in the texture unit, reusing the original texture cache, increasing the flexibility of the filtering unit, and packing the input data and weight parameters to fixed-point format, we make the texture unit be able to support convolutional and pooling layers with only small modifications. The proposed architecture is verified by integrating TTU into a GPU system in RTL level. Experimental results show that 18.54x speedup can be achieved with the overhead of only 8.5% compared with a GPU system with a traditional texture unit.

In the field of approximate nearest neighbor (ANN) search, rare of the existing approaches are tailored for video applications. The Ring Intersection Approximate Nearest Neighbor (RIANN) is the first ANN search algorithm for videos. It achieves real-time by performing the ANN search on the sparse grid and interpolating others. For some applications, the dense ANN search is needed to ensure the searching accuracy. To achieve dense ANN search in real-time, we consider the parallel computing as a solution. However, the RIANN algorithm is not suitable for parallel computing as the algorithm itself suffers from bad thread coherency. In this paper, we propose the Sphere Ring Intersection Approximate Nearest Neighbor (SRIANN), which solves the problem of bad thread coherency and improves the accuracy of ANN search compared to the original RIANN method. The experimental results show that the proposed method is the only one able to perform dense ANN search for CIF videos in real-time.

Convolutional neural networks (CNNs) have emerged to provide powerful discriminative capability, especially in the field of image recognition and object detection. However, their massive computation requirements, storage and memory accesses make them hard to be deployed on mobile or embedded systems. In this paper, a few optimizations for a CNN cascade face detection algorithm are proposed to increase throughput while minimizing computation, storage and bandwidth requirement under power constraints. First, the first net of the CNN cascade is converted to a fully convolutional network to reduce 83% of the computation. Second, network retraining is applied to quantize the model parameters from 32-bit floating-point to 2-bit fixed-point, resulting in 93.75% less parameter memory size. Finally, a CNN accelerator is implemented on a Xilinx ZYNQ FPGA board.

Saliency detection, or salient object detection, is an essential pre-processing step for many computer vision applications. It extracts the most conspicuous part of an image and reduces the computation and transmission requirement. This ability is desired for end devices with limited hardware resources. However, existing algorithms are not suitable for hardware implementation. Traditional works usually build upon manually designed priors, and their computations usually involve irregular memory access. Recently, deep learning based algorithms have demonstrated superior performance, while they require a large number of parameters and computation. In this paper, we propose a hardware-efficient algorithm for salient object detection. Our algorithm first uses a lightweight CNN to predict a coarse saliency map, which is then refined to obtain the boundary-accurate saliency map. We demonstrate that our two-stage algorithm can achieve favorable performance compared to existing methods while being more hardware-efficient regarding computation and memory requirement.

Vehicle re-identification (Re-ID) is fundamentally challenging due to the difficulties in data labeling, visual domain mismatch between datasets and diverse appearance of the same vehicle. We propose the adaptive feature learning technique based on the space-time prior to address these issues. The idea is demonstrated effectively in both the human Re-ID and the vehicle Re-ID tasks. We train a vehicle feature extractor in a multi-task learning manner on three existing vehicle datasets and fine-tune the feature extractor with the adaptive feature learning technique on the target domain. We then develop a vehicle Re-ID system based on the learned vehicle feature extractor. Finally, our meticulous system design leads to the second place in the 2018 NVIDIA AI City Challenge Track 3.

Superpixel segmentation has been widely used in many computer vision tasks. Existing superpixel algorithms are mainly based on hand-crafted features, which often fail to preserve weak object boundaries. In this work, we leverage deep neural networks to facilitate extracting superpixels from images. We show a simple integration of deep features with existing superpixel algorithms does not result in better performance as these features do not model segmentation. Instead, we propose a segmentation-aware affinity learning approach for superpixel segmentation. Specifically, we propose a new loss function that takes the segmentation error into account for affinity learning. We also develop the Pixel Affinity Net for affinity prediction. Extensive experimental results show that the proposed algorithm based on the learned segmentation-aware loss performs favorably against the state-of-the-art methods. We also demonstrate the use of the learned superpixels in numerous vision applications with consistent improvements.

In this paper, we review the design of Minimum Barrier Distance and propose a new path-wise distance metric called Minimum Spanning Distance (MSD). Unlike most existing distance metrics, which only define distance between two pixels on gray-scale images, the proposed distance metric conceptually estimates the color space spanned by the colors on the path of interest. Therefore, the MSD takes into consideration the three channels on color images at the same time to compute distance. Compared with other distance metrics, MSD can not only achieve the highest numerical scores but also produce visually good segmentation maps in our experiment of interactive segmentation on the Gulshan dataset.

Deep Convolutional Neural Networks (CNNs) are widely employed in modern computer vision algorithms, where the input image is convolved iteratively by many kernels to extract the knowledge behind it. However, with the depth of convolutional layers getting deeper and deeper in recent years, the enormous computational complexity makes it difficult to be deployed on embedded systems with limited hardware resources. In this paper, we propose two computation-performance optimization methods to reduce the redundant convolution kernels of a CNN with performance and architecture constraints, and apply it to a network for super resolution (SR). Using PSNR drop compared to the original network as the performance criterion, our method can get the optimal PSNR under a certain computation budget constraint. On the other hand, our method is also capable of minimizing the computation required under a given PSNR drop., This paper was accepted by 2018 The International Symposium on Circuits and Systems (ISCAS)

In this paper, we present a physics filter which can enhance the user experience of human computer interaction applications. We demonstrate the filter through an interaction system with head gestures and movements. Users interact with the computer through the wink detection function, implemented by eye brightness intensity analysis, and head tracking function, implemented by a facial landmark detector. However, both intensity and facial landmark data are unstable in temporal domain, which deteriorates user experience. Our proposed physics filter is capable of enhancing stability with minimizing latency. This filter can be used in various interaction applications where data stabilization is required.

Accurately tracking the six degree-of-freedom pose of an object in real scenes is an important task in computer vision and augmented reality with numerous applications. Although a variety of algorithms for this task have been proposed, it remains difficult to evaluate existing methods in the literature as oftentimes different sequences are used and no large benchmark datasets close to realworld scenarios are available. In this paper, we present a large object pose tracking benchmark dataset consisting of RGB-D video sequences of 2D and 3D targets with ground-truth information. The videos are recorded under various lighting conditions, different motion patterns and speeds with the help of a programmable robotic arm. We present extensive quantitative evaluation results of the state-of-the-art methods on this benchmark dataset and discuss the potential research directions in this field. The proposed benchmark dataset is available online at media.ee.ntu.edu.tw/research/OPT.

Recently, convolutional neural networks (CNNs) have achieved great success in fields such as computer vision, natural language processing, and artificial intelligence. Many of these applications utilize parallel processing in GPUs to achieve higher performance. However, it remains a daunting task to optimize for GPUs, and most researchers have to rely on vendor-provided libraries for such purposes. In this paper, we discuss an operator that can be used to succinctly express computational kernels in CNNs and various scientific and vision applications. This operator, called Unrolled-Memory-Inner-Product (UMI), is a computationally-efficient operator with smaller code token requirement. Since a naive UMI implementation would increase memory requirement through input data unrolling, we propose a method to achieve optimal memory fetch performance in modern GPUs. We demonstrate this operator by converting several popular applications into the UMI representation, and achieve 1.3x-26.4x speedup against frameworks such as OpenCV and Caffe.

In this study, we present a set of new evaluation measures for the track-based multi-camera tracking (T-MCT) task leveraging the clustering measurements. We demonstrate that the proposed evaluation measures provide notable advantages over previous ones. Moreover, a distributed and online T-MCT framework is proposed, where re-identification (Re-id) is embedded in T-MCT, to confirm the validity of the proposed evaluation measures. Experimental results reveal that with the proposed evaluation measures, the performance of T-MCT can be accurately measured, which is highly correlated to the performance of Re-id. Furthermore, it is also noted that our T-MCT framework achieves competitive score on the DukeMTMC dataset when compared to the previous work that used global optimization algorithms. Both the evaluation measures and the inter-camera tracking framework are proven to be the stepping stone for multi-camera tracking.

In a remote rendering system, the display device and the main graphics processing unit (GPU) are located in different places, which is a kind of client-server architecture and is widely used in cloud gaming and virtual reality (VR). To reduce the interaction delay and improve the user experience especially when the link between the server and client becomes wireless, we propose a client-end GPU-accelerated scene warping technique to approximate the rendered frames between key frames, meanwhile hiding the interaction delay. The distributed rendering technique can warp and interpolate images on the client end with the server-rendered reference background images and the corresponding depth maps. A mobile GPU is also employed to accelerate the image warping operation. In addition, the foreground layer is rendered with GPU and then blended with the approximated background layer generated from background warping. A prototype of the proposed system is implemented with a commercialized smart phone. Compared with the video streaming-based cloud gaming system, the evaluation results show that the Game Mean Opinion Score (GMOS) increases from 1 to 4.1 for the interactive games even in a high network delay condition.

In order to address the high transmission bandwidth requirement of an Internet-of-Video-Things (IoVT), an object-based on-line video summarization algorithm is proposed to summarize the captured video information at the sensor nodes before being transmitted to the server. It is composed of two stages: intra-view and inter-view stages. In the intra-view stage, human object detector is employed with the proposed human object descriptor. In the inter-view stage, an on-line clustering algorithm with a two-layer K-nearest-neighbor model is also proposed for object clustering. Experimental results show that significant improvement can be achieved when compared with state-of-the-art works.

Bilateral filtering (BLF) and median filtering (MF) are key components in many applications. As the image resolution grows rapidly, implementation of efficient filtering is highly demanded. In this paper, we present a unified VLSI architecture that is able to compute both kinds of filters for 4k2k videos at 30fps. One feature of this design is that we leverage an emerging layer-based algorithm for both BLF and MF, which turns the non-linear BLF or MF into a set of linear filtering followed by output interpolation. It allows more accurate approximation than previous works. Moreover, to address the introduced design challenges of filter units and comparison units, we propose an architecture with moving sum method (MSM) of box filtering and a parallel component selector (PCS), respectively. The MSM requires only one line buffer, and the PCS architecture allows us to complete the comparison in one cycle. Furthermore, a shared pre-computation unit is also proposed to support both kinds of filters. As a result, our design costs only 6.33KB on-chip memory and 161.5K logic gates to support both BLF and MF. It is much efficient than existing architectures which support BLF only.

In this paper, a distributed video coding (DVC) system with spatiotemporal side information is proposed. The proposed framework addresses the problem of poor compression performance of DVC for high-motion video sequences by integrating temporal and spatial prediction schemes in one framework. Super-resolution techniques are employed for spatially-predicted side information generation, and a support vector machine is trained to adaptively select the coding structure with both spatial and temporal prediction. In addition, an encoder-driven coding mode selection at different granularities, including frame, block and coefficient levels, is adopted to further improve the coding performance for various video conditions. Experimental results show that the average BD rate reduction of the proposed framework is 12.93% compared with the DISCOVER DVC, and the coding gain is significant, especially for high-motion sequences. Moreover, the average computing complexity is only 92.26% of the DISCOVER DVC.

Real-time recovering an accurate 6 DoF pose of a known planar target is essential for augmented reality and robotics applications. Despite several pose estimation tracking systems have been proposed over recent years, there is still the need for a more efficient and more accurate solution for general planar objects. In this work, we develop an innovative GPU implementation of a real-time pose estimation and tracking system. It consists of a pose estimation unit and a pose tracker unit. While the former computes an initial pose of a target using direct method, the latter realizes accurate pose tracking with a hierarchical search scheme. Experiments on both synthetic and real datasets demonstrate that the proposed algorithm performs favorably with various planar targets. By implementing our method on an embedded GPU, the system achieves to work at 11 FPS and is suitable for real-time applications.

Visualization of the 3D point cloud model has played a crucial part in augmented reality and virtual reality for a long time. Besides, the technique of modeling also greatly benefit the application related to understanding crowd dynamics. In this paper, we introduce a technique to restore the the background area occluded by the front objects from a single viewpoint. In our procedure, we adopt the idea of exemplar-based in-painting and the surface detection method. We first detect the plane in the 3D point cloud model to help recover the depth map of the scene and reconstruct the geometric structure behind our target object. Further, we decide the colors of the hole on background which is resulted from the removal of the foreground object. At last, we rebuild the scene according to the inpainted RGB and depth images to achieve a more well-rounded visualization. We present several indoor examples with satisfying results.

Bilateral filtering is a commonly used technique in image processing. However, being nonlinear, it is computationally expensive. The situation gets worse while the filter radius grows up. Several works have been proposed to accelerate the computation. Nevertheless, most techniques are tailored for grayscale image bilateral filtering or confined to specific kernel functions. In this paper, we propose a constant time bilateral filter for color images. Specifically, we extend an existing constant time bilateral filtering technique, which has been demonstrated the state-of-the-art for gray-scale images. We generalize the original approach as a three-stage algorithm. Based on the generalization, we explain the drawbacks of its naive extension for color images and propose a solution that adapts to the image content. Experimental results demonstrate the effectiveness of our solution.

With the goal of increasing the resolution of face images, recent face hallucination methods advance learning techniques which observe training low and high-resolution patches for recovering the output image of interest. Since most existing patch-based face hallucination approaches do not consider the location information of the patches to be hallucinated, the resulting performance might be limited. In this paper, we propose an anchored patch-based hallucination method, which is able to exploit and identify image patches exhibiting structurally and spatially similar information. With these representative anchors observed, improved performance and computation efficiency can be achieved. Experimental results demonstrate that our proposed method achieves satisfactory performance and performs favorably against recent face hallucination approaches.

Color transfer is an image processing technique commonly used to fix images with wrong colors, enhance the lighting conditions, or produce special styles to express specific emotions. With the aid of a reference image, the intended color characteristics can be properly transferred to the source images or videos. Since the applications start growing popular, many image color transfer methods have emerged; However, only few video color transfer algorithms have been proposed so far. In this paper, we propose an efficient example-based color transfer algorithm for both images and videos, which preserves the gradient details of the source by building Laplacian pyramids and improves the spatio-temporal consistency by employing patch matching method. Other than the existing quality metrics, we further propose a video quality metric regarding temporal consistency and compare our algorithm with other color transfer methods. The experimental results show that our algorithm generally produces outputs with high fidelity in terms of colors, scene details, and also spatiotemporal consistency.

Wearable social camera is an egocentric camera that summarizes the video of the user's social activities. This paper proposes a core technology of the wearable social camera: egocentric video summarization for social interaction. Different from other works of third-person action/interaction recognition in egocentric videos, which focus on distinguishing different actions, this work finds the common features among all the interactions, which is called interaction features (IF). IF of the third-person is proposed to be composed of three parts: physical information of head, body languages, and emotional expression. Furthermore, hidden Markov model (HMM) is employed to model the interaction sequences, and a summarized video is generated with hidden Markov support vector machine (HM-SVM). Experimental results with a life-log dataset show that the proposed system performs well for summarizing life-log videos.

Face hallucination technique generates high-resolution face images from low-resolution ones. In this paper, we propose a patch based multitask deep learning method for face hallucination, which is robust to blurring of images. Our method is based on fully connected feedforward neural network, and the weights of the final layers are fine-tuned separately on different clusters of patches. Experimental results show that our system outperforms the prior state-of-the-art methods by a significant margin, while using less testing computation time.

In this paper, a perceptual video coding system is developed based on the newest coding standard-HEVC/H.265. We propote a local Just-Noticeable-Difference (JND) model, which simultaneously considers the contrast sensitivity model, luminance masking, temporal masking, residue-dependent adjustment, and the summation effect of quantization distortion to get the weighting of importance of human eye perception for each coding unit (CU) in a video frame. The proposed algorithm achieves better bit allocation for video coding systems by changing quantization parameters at CU level. Simulation and subjective experiments show that our system achieves average 14% bitrate saving in the QP range of 27–37 without perceptual quality degradation.

Estimating 3D pose of a known object from a given 2D image is an important problem with numerous studies for robotics and augmented reality applications. While the state-of-the-art Perspective-n-Point algorithms perform well in pose estimation, the success hinges on whether feature points can be extracted and matched correctly on targets with rich texture. In this work, we propose a robust direct method for 3D pose estimation with high accuracy that performs well on both textured and textureless planar targets. First, the pose of a planar target with respect to a calibrated camera is approximately estimated by posing it as a template matching problem. Next, the object pose is further refined and disambiguated with a gradient descent search scheme. Extensive experiments on both synthetic and real datasets demonstrate the proposed direct pose estimation algorithm performs favorably against state-of-the-art feature-based approaches in terms of robustness and accuracy under several varying conditions.

As better performance is achieved by deep convolutional network with more and more layers, the increasing number of weighting and bias parameters makes it only possible to be implemented on servers in cyber space but infeasible to be deployed in physical-world embedded systems because of huge storage and memory bandwidth requirements. In this paper, we proposed an efficient method to quantize the model parameters. Instead of taking the quantization process as a negative effect on precision, we regarded it as a regularize problem to prevent overfitting, and a two-stage quantization technique including soft- and hard-quantization is developed. With the help of our quantization method, not only 93.75% of the parameter memory size can be reduced by replacing the word length from 32-bit to 2-bit, but the testing accuracy after quantization is also better than previous approaches in some dataset, and the additional training overhead is only 3% of the ordinary one.

Gaze tracking systems have high potential to be used as natural user interface devices; however, the mainstream systems are designed with infrared illumination, which may be harmful for human eyes. In this paper, a real-time eye localization, blink detection, and gaze estimation system is proposed without infrared illumination. To deal with various lighting conditions and reflections on the iris, the proposed system is based on a continuously updated color model for robust iris detection. Moreover, the proposed algorithm employs both the simplified and the original eye images to achieve the balance between robustness and accuracy. Experimental results show that the proposed system can achieve the accuracy of 96.8% for blink detection and the accuracy of 1.973 degree for gaze estimation with the processing speed of 10–11fps. The performance is comparable to previous works with infrared illumination.

Image denoising is always an active problem in image processing over the years. Though the principles to gray-scale image denoising have been extensively explored in recent advances. Relatively few works focus on the problem of color image denoising. In this paper, we address the denoising problem for natural color images. We propose a simple yet effective denoising algorithm based on the local color line assumption for natural color images. We further show that the proposed denoising strategy can be easily realized by slight modification to the guided filter. Moreover, the intermediate data in guided filter computation is utilized to make the denoising algorithm highly adaptable to local noise level. Experimental results show that the proposed method is effective to natural color image denoising and comparable to other complex methods with low complexity.

In order to reduce the required transmission bandwidth of a video sensor network for Internet-of-Things applications, a video sensor node with distributed video summary capability is developed, where the video summarization algorithm is developed based on a scene model and a frame color descriptor. Experimental results show that by exchanging information between video sensors, the required transmission bandwidth is drastically reduced. Moreover, real-time performance is achieved on an embedded platform.

In this work, a first time realization of an on-line LMD processor is proposed. The processor chip is implemented with the core area of 2.33mm2 using TSMC 90-nm process with high-Vt cell. Operating at 20MHz, the chip consumes 2.28mw from a 1.0V supply voltage. Confirmed by post-layout simulation, after the initial latency of 0.833 seconds, the proposed processor is able to produce decomposed ECG data of 5 product functions and a residue in short burst every 277.8ms with computational latency 4.9ms. The accuracy of the hardware design is validated through decomposing real ECG data obtained from MIT-BIH arrhythmia database. The performance shows that the proposed on-line LMD processor is well suited to low-power, real-time wearable biomedical applications.

Painted face effect, where the textured contents are projected on the presenter's face, frequently occurs during presentations with projectors. To remove this annoying effect, in this paper, a projector-camera system is designed to modify the projection content according to the location information of the presenter generated with a background segmentation technique. A new non-linear photometric model for a projector-camera system is also introduced as well as an ambient light adaptation scheme to generate the background model for object segmentation under varied lighting conditions. Experimental results show that this system can successfully remove painted face effect without introducing any artifacts. This technique can improve the user experience for presentations with projectors.

In application of human computer interaction, modeling of human vision is an approach seems appropriate but has not yet been accomplished. The main contribution in this paper is proposing an intuitive system for human computer interaction by modeling human vision using human sight position and human body part position, as well as an evaluation method to estimate the characteristic of the proposed system.

Nowadays, 3D scene reconstruction using RGB-D videos becomes more popular because of the widely-available off-the-shelf RGB-D camera. However, the depth information from current RGB-D camera still need improved in order to reconstruct the 3D scene with better quality. In this paper, an edge-aware depth completion method aims to recover more accurate depth information is proposed. There are mainly two parts in our proposed method. The first part is the edge-aware color image analysis, and the second part is depth image processing including unreliable depth pixel invalidation and filling. The depth image processing can retrieve more accurate depth information using our proposed edge-aware color image analysis. Consequently, we can not only preserve the reliable depth information, but also fill in the appropriate depth values to align edges of depth image with edges of its corresponding color image. Besides, the experimental results show that the visualization of the reconstructed point-cloud 3D scene benefits from our proposed edge-aware depth completion. Finally, the PSNR evaluation using ground truth depth information is presented.

Video sensors are widely used in many applications such as security monitoring and home care. However, the growth of the number of sensors makes it impractical to stream all videos back to a central server for further processing, due to communication bandwidth and server storage constraints. Multi-view video summarization allows us to discard redundant data in the video streams taken by a group of sensors. All prior multi-view summarization methods, however, process video data in an off-line and centralized manner, which means that all videos are still required to be streamed back to the server before conducting the summarization. This paper proposes an on-line, distributed multi-view summarization system, which integrates the ideas of Maximal Marginal Relevance (MMR) and MS-Wave, a bandwidth-efficient distributed algorithm for finding k-nearest-neighbors and k-farthest-neighbors. Empirical studies show that our proposed system can discard redundant videos and keep important keyframes as effectively as centralized approaches, while transmitting only 1/6 to 1/3 as much data.