Search

In April 2016, a massive earthquake with a magnitude of 7.3 struck Kumamoto region, Japan, causing major devastation. One of the structures that were damaged in Kumamoto was Kumamoto Castle, a cultural asset of great significance in Japan. The stone retaining wall “ishigaki” that formed the foundation of the castle collapsed, and the superstructure was destroyed. The number of stones is estimated to be more than 70,000, and restoration work is anticipated to take more than 20 years. Since each of the stones is an important cultural asset, the broken stone structure needed to be restored to its original state in order not to lose its cultural value forever. In addition, the fallen stones need to be returned to their original positions in the ishigakis. In similar cases, non-automatic visual verification was used. However, for Kumamoto Castle, this would have been impossible because a large number of stones were displaced as a result of the collapse. The purpose of this project is to provide support for the restoration work by matching the stones that fell down after the collapse with those before the collapse using information technology, such as computer vision and optimization technologies. Specifically, we captured photographic images of the stones before and after the collapse to match them. The technical contributions of this study are as follows: (a) To estimate the scale and surface orientation of the stones, we exploit 3D model construction from the images. (b) To solve the jigsaw-puzzle-like problem of reassembling the stone fragments, we exploit the combination of a customized iterative closest point (ICP) algorithm for shape position matching and an assignment algorithm to find the best pairs of stones before and after the collapse by using the matching degree obtained from ICP. Here, only the 2D shape of the stones before the collapse can be used due to the small number of photos available. In contrast, a detailed 3D shape can be obtained from the stones after their collapse. We matched these asymmetric data in 2D and 3D to enable a comprehensive reconstruction. (c) We developed a user-friendly graphical user interface system that was used by actual masons without special knowledge. The developed system was used to match the ishigaki of a turret, Iidamaru. As a result, we succeeded in identifying 337 stones, or approximately 90% of the 370 images. These results are expected to be useful for and were used as a blueprint during actual restoration work.

When learning to play a musical instrument, it is important to improve the quality of self-practice. Many systems have been developed to assist practice. Some practice assistance systems use special sensors (pressure, flow, and motion sensors) to acquire the control parameters of the musical instrument, and provide specific guidance. However, it is difficult to acquire the control parameters of wind instruments (e.g., saxophone or flute) such as flow and angle between the player and the musical instrument, since it is not possible to place sensors into the mouth. In this paper, we propose a sensorless control parameter estimation system based on the recorded sound of a wind instrument using only machine learning. In the machine learning framework, many training samples that have both sound and correct labels are required. Therefore, we generated training samples using a robotic performer. This has two advantages: (1) it is easy to obtain many training samples with exhaustive control parameters, and (2) we can use the correct labels as the given control parameters of the robot. In addition to the samples generated by the robot, some human performance data were also used for training to construct an estimation model that enhanced the feature differences between robot and human performance. Finally, a flute control parameter estimation system was developed, and its estimation accuracy for eight novice flute players was evaluated using the Spearman’s rank correlation coefficient. The experimental results showed that the proposed system was able to estimate human control parameters with high accuracy.

In this paper, we propose a pseudo long short-term memory (LSTM) classifier for single image vehicle classification. The proposed pseudo-LSTM (P-LSTM) uses spatially divided images rather than time-series images. In other words, the proposed method considers the divided images to be time-series frames. The divided images are formed by cropping input images using two-level spatial pyramid region configuration. Parallel convolutional networks are used to extract the spatial pyramid features of the divided images. To explore the correlations between the spatial pyramid features, we attached an LSTM classifier to the end of the parallel convolutional network and treated each convolutional network as an independent timestamp. Although LSTM classifiers are typically used for time-dependent data, our experiments demonstrated that they can also be used for non-time-dependent data. We attached one fully connected layer to the end of the network to compute a final classification decision. Experiments on an MIO-TCD vehicle classification dataset show that our proposed classifier produces a high evaluation score and is comparable with several other state-of-the-art methods.

Two strong earthquakes that hit Kumamoto in April 2016 damaged the infrastructure and many buildings in the area, significantly impacting the people and traditional cultural assets of Kumamoto. In this project, using information technology, we aim to reconstruct the stone walls of Kumamoto castle, which is an important nationally designated special historical site in Japan. A stone wall is called "Ishigaki'' in Japanese. During the earthquake, many of the stones of the Kumamoto castle wall fell to the ground. Therefore, to reconstruct the stone wall, we must identify the original locations of the collapsed stones, similar to a jigsaw puzzle. In this paper, we formulate the problem by defining the cost function of the similarity between before and after collapse of the stones and propose the application of a combinatorial optimization algorithm. We tested the proposed solver by identifying 269 actual collapsed stones of the castle wall, and the results demonstrated that the proposed method can identify 253 stones correctly (94% accuracy).

In this paper, we describe the fabrication and control of a robot that plays an acoustic guitar using a solenoid. So far, various robots that play guitars have been produced, but the robot produced in this research are characterized in that they only perform linear movement of solenoids and perform the fretting and picking operations in playing guitars. Also, by adjusting the operation of the solenoid by PWM control, the strength and weakness of the sound can be expressed. Furthermore, by adopting serial communication based on the MIDI between the robot and the PC, the user can easily control the robot. This time, we aimed to make the sound strong and weak with open strings. As a result of the experiment, it was found that in the open string, the sound strength change can be applied to all the strings. As future points of improvement, it was mentioned to be able to pick up well even if the height of the string is changed by pushing strings, and to further reduce the noise generated between the parts of the solenoid and the robot.

In this paper, we propose a new framework for generating big sized dataset using synthetic data generation by robotics. In learning-based recognition, for example, using convolutional neural networks (CNNs), it is critical for the performance, to collect high-quality and large amounts of training data. Previously, to increase the training data set, a data augmentation technique based on digital signal processing were applied to the original sound data. However, the data augmentation based on digital signal processing data is a limited method, because it depends on some previous knowledge of the data and cannot perform for all domains. On the other hand, we propose a new dataset collection technique using a robot that automatically plays instruments, by which it becomes possible to add high-quality data to training samples. Experimental results for guitar chord recognition show that the proposed method using CNNs and a guitar robot can outperform the CNN systems with the traditional data augmentation.

Zoetrope is an animation display that produces the illusion of motion via a displayed sequence of pictures; this device was invented in 1833. The traditional Zoetrope consists of a rotational disk and a strobe light. The disk has some plates that are pasted together showing sequential pictures and is rotated rapidly. The strobe light is emitted with synchronization to the rotation angle To display the animation. The Zoetrope has been improved and there are many kinds of the zoetropes in SIGGRAPH community: interactive zoetrope [Smoot et al. 2010], ZoeMatrope [Miyashita et al. 2016], and Magic Zoetrope [Yokota and Hashida 2018].

This paper proposes multi-view display using a digital light processing (DLP) projector and new active shutter glasses. In conventional stereoscopic active shutter systems, active shutter glasses have a 0--1 (open and closed) state, and the right and left frames are temporally divided. However, this causes the display to flicker because the human eye perceives the appearance of black frames when the other shutter is closing. Furthermore, it is difficult to increase the number of views because the number of frames representing images is also divided. We solve these problems by extending the active shutter beyond the use of the 0--1 state to a continuous range of states [0, 1] instead. This relaxation leads to the formulation of a new DLP imaging model and an optimization problem. The special structure of DLP binary imaging and the continuous transmittance of the new active shutter glasses require the solution of a binary continuous image decomposition problem. Although it contains NP-hard problems, the proposed algorithm can efficiently solve the problem. The implementation of our imaging system requires the development of an active shutter device with continuous transmittance. We implemented the control of the transmittance of the liquid crystal display (LCD) shutter by using a pulse-width modulation (PWM). A simulation and the developed multi-view display system were used to show that our model can represent multi-view images more accurately than the conventional time-division 0-1 active shutter system.

Projectors using digital mirror devices (DMDs) are widely used in consumer electronics. DMDs can switch between the ON and OFF states. Light is reflected by the DMDs, and a binary image is projected onto a screen. Full-color images can be represented by switching 24-binary images rapidly. To increase the frame rate, it is necessary to speed up the DMD switching. However, this is difficult because the switching time is limited by the mechanical constraints of the DMD. One solution is reducing the number of binary images to represent a full-color image. This paper proposes a framework for optimizing the color quantization and light-emitting diode (LED) luminance. The constructed system was evaluated by using a projector and complementary metal–oxide–semiconductor (CMOS) camera. The experimental results indicated that the proposed method improves the total luminance of a projected image approximately 122% compared to previous models and produces better image quality1.

In recent years, many researchers have shown increasing interest in music information retrieval (MIR) applications, with automatic chord recognition being one of the popular tasks. Many studies have achieved/demonstrated considerable improvement using deep learning based models in automatic chord recognition problems. However, most of the existing models have focused on simple chord recognition, which classifies the root note with the major, minor, and seventh chords. Furthermore, in learning-based recognition, it is critical to collect high-quality and large amounts of training data to achieve the desired performance. In this paper, we present a multi-task learning (MTL) model for a guitar chord recognition task, where the model is trained using a relatively large-vocabulary guitar chord dataset. To solve data scarcity issues, a physical data augmentation method that directly records the chord dataset from a robotic performer is employed. Deep learning based MTL is proposed to improve the performance of automatic chord recognition with the proposed physical data augmentation dataset. The proposed MTL model is compared with four baseline models and its corresponding single-task learning model using two types of datasets, including a human dataset and a human combined with the augmented dataset. The proposed methods outperform the baseline models, and the results show that most scores of the proposed multi-task learning model are better than those of the corresponding single-task learning model. The experimental results demonstrate that physical data augmentation is an effective method for increasing the dataset size for guitar chord recognition tasks.

Cross-modal retrieval is a hot topic in the fields of machine learning and media retrieval, making it possible to relate different types of media, such as image, text, and audio. A powerful method for the cross-modal retrieval, discrete cross-modal hashing (DCH), has recently been proposed. The DCH can encode different types of media feature vectors to binary codes. When stored in a database, the binary code makes searches efficient because the Hamming distance between the corresponding sections of two binary codes can be computed via a specialized CPU operation. Moreover, it has recently been shown that when optimizing hash functions for supervised discrete hashing (SDH), Hadamard matrices can be used, and this technique is named “HC-SDH”. In this study, we apply the HC-SDH to cross-modal hashing. Experimental results demonstrate that the proposed cross-modal hashing can achieve the same performance as the conventional DCH with 1/24 of the training time.

In this paper, we proposed a lightweight spatial pyramid convolutional neural network (SP-CNN) classifier for image-based traffic sign classification. The lightweight SP-CNN classifier is formed based on ResNet (residual network) CNN architecture which originally used for CIFAR10 image classification problems. Our proposed classifier consists of five parallel convolutional networks and each network processes a cropped region using spatial pyramid configuration. For smoother transitions between the regions cropped in the level 1 of spatial pyramid configuration, we overlap the level 1 of spatial pyramid regions configuration for around 12.5% on each axis. The proposed classifier trained by fine-tuning the CIFAR10 weights with NAG (Nesterov Accelerated Gradient) training algorithm. Experiments on GTSRB (German Traffic Sign Recognition Benchmark) dataset show that our lightweight SP-CNN version produces an accuracy of 99.70% and an execution time of 60 ms. The proposed classifier produces a very competitive accuracy compared with other methods but with less number of parameters.

In this paper, we propose a learning-based supervised discrete hashing method. Binary hashing is widely used for large-scale image retrieval as well as video and document searches because the compact binary code representation is essential for data storage and reasonable for query searches using bit-operations. The recently proposed supervised discrete hashing (SDH) method efficiently solves mixed-integer programming problems by alternating optimization and the discrete cyclic coordinate descent (DCC) method. Based on some preliminary experiments, we show that the SDH method can be simplified without performance degradation. We analyze the simplified model and provide a mathematically exact solution thereof; we reveal that the exact binary code is provided by a "Hadamard matrix." Therefore, we named our method Hadamard codedsupervised discrete hashing (HC-SDH). In contrast to SDH, our model does not require an alternating optimization algorithm and does not depend on initial values. HC-SDH is also easier to implement than iterative quantization (ITQ). Experimental results involving a large-scale database show that Hadamard coding outperforms conventional SDH in terms of precision, recall, and computational time. On the large datasets SUN-397 and ImageNet, HC-SDH provides a superior mean average of precision (mAP) and top-accuracy compared to the conventional SDH methods with the same code length and FastHash. The training time of HC-SDH is 170 times faster than conventional SDH and the testing time including the encoding time is seven times faster than FastHash which encodes using a binary-tree.

In this paper, we proposed a hierarchical spatial pyramid pooling mechanism for improving fine-grained vehicle classification problems. Our proposed method created by removing the last layer of convolutional neural network (CNN) classifier, attaching the hierarchical spatial pyramid pooling at the end of the network, and fine-tune the CNN classifier. Our hierarchical spatial pyramid pooling consists of two independent spatial pyramid pooling that pooled the features from two different layers in the original CNN classifier. We modified ResNet50 and AlexNet CNN classifier using our proposed method and test it on fine-grained vehicle classification dataset. Unlike part-based classifier, which very popular method to tackle fine-grained classification problems, our method does not require any pre-processing or post-processing step in order to do the classification tasks. Experiments on BoxCars fine-grained vehicle classification dataset shows that our proposed method can increase the overall accuracy of the classifier and some of the results achieve state-of-the-art performance.

Developed shogi robot system The authors developed a low-cost, safety shogi robot system. A Web camera installed on the lower frame is used to recognize pieces and their positions on the board, after which the game program is played. A robot arm moves a selected piece to the position used in playing a human player. A fast, robust image processing algorithm is needed because a low-cost wide-angle Web camera and robot are used. The authors describe image processing and robot systems, then discuss experiments conducted to verify the feasibility of the proposal, showing that even a low-cost system can be highly reliable.

A speedy filter for removing impulse noise based on an adaptive window observation is presented. The proposed method not only reduces specific content of impulse noise but also produces a low computational complexity in several noise densities, which exploits the adaptive concept. In this research, we use an adaptive window. It is a combination of the post filtered pixel and the pixel that would be filtered. Based on the adaptive window observation, we can get the output filter. The numerical results of using peak signal-to-noise ratio and computational time prove that the proposed method is able to reduce impulse noise and to produce a low computational complexity. Furthermore, the proposed method is capable of overcoming the drawback of previous studies and provides a satisfactory result.

The demand of High Definition Maps (HD-Maps) has been increasing, especially for autonomous vehicle application. Usually, HD-Map is created by scanning the road using LiDAR sensor and reconstructing the road on 3D world to capture all aspects of road properties. One of the important properties of a road is its edge or boundary. In this paper, we propose end-to-end 3D Encoder-Decoder Convolutional Network (3D-EDCN) for road edge detection on 3D point cloud data produced by LiDAR sensor. Our 3D-EDCN classifier consists of nine convolutional layers and three deconvolutional layers. For simplification, we use 3D voxel format as input and output of the classifier. Our proposed method was tested using our own 3D point cloud dataset which taken from LiDAR equipment and consisting of 103 3D point cloud data with their respective road edge ground truth. In the training process, we use combinations of Cross-Entropy loss function and Euclidean loss function to help our model converged. As a preliminary result, our proposed 3D-EDCN classifier achieves Mean Square Error (MSE) of 2.738×10−5, precision of 0.37262, and recall of 0.14432.

Recently, welfare vehicles are widely used among the elderly and the aged. However, it is not easy for inexperienced aged to control the welfare vehicle in narrow places such as elevators and narrow corridors. In this paper, driving assistance system for the welfare vehicle which is controlled based on Augmented Reality (AR) is proposed. It is supposed that cognitive ability from the objective viewpoint is enhanced. First, a concept of the virtual platoon control with a virtual preceding vehicle and a real following vehicle is outlined; a virtual preceding vehicle projected on Head Mounted Display (HMD) is steered virtually with an objective viewpoint, and the real vehicle that the aged rides on is followed so as to drive without collision with obstacles. Finally, the proposed control system is assessed by trial runs using the welfare vehicle STAVi, and the effectiveness using the assistance system with AR is evaluated by a driving experiment in narrow spaces.

In this study, we propose a method for designing color quantized images for use with digital micromirror device (DMD-based projectors. It is necessary to quantize full-color images if we want to achieve a high frame rate in DMD-based systems. The quantized image should be similar to a full-color image to achieve good results. The proposed method for designing quantized images is based on an evaluation function and a particle swarm optimization algorithm. We can improve the image quality and luminance of the image displayed by tuning the weight values of the evaluation function. We evaluated the quality of the designed binary images using numerical examples.

Batik is one of the cultural heritages of Indonesia that have many different motifs in each region as well as in its usage. However, the Indonesians sometimes not knowing the batik motif that they’re wearing every day, and sometimes they have a batik image without knowing batik information contained in their batik image. With the growing number of images of batik and batik motifs, a classification method that can classify various motifs of batik is required to automatically detect the motif from the batik image. Image processing using the Deep Learning especially for image classification is widely used recently because it has good results. The most popular method in deep learning is Convolutional Neural Network (CNN) which has been proved robust in natural images. This study offers a batik motif image classification system using CNN method with new network architecture developed by combining GoogLeNet and Residual Networks named IncRes. IncRes merges the Inception Module with Residual Network structure. With the 70.84% accuracy, the system can be used to classify the batik image motif accurately.

Shared human actions in the video are the biggest problem for video classification system. For example, long jump sports video will share a running action with the long jump or running sports video. In this paper, we present a video classification system by combining the keyframe extractor system and convolutional neural network (CNN) classifier. The visual attention modeling was used to build the keyframe extractor system and top k frames with the highest saliency value is chosen for the classification process. By using the top k keyframe with the highest saliency value, it may reduce the shared action of the video and makes the classifier easier to classify the video by using only the spatial features. The keyframe extracted from video summarization method was used for training process, which in our system proved very efficient and speed up the training process. As a result, our system is effective and the average accuracy is increased compared with the system without using the keyframe extractor system. Our proposed method also outperforms the system using video summarization method as keyframe extractor system by around 3%.

Using image processing to extract nodular or linear shadows is a key technique of computer-aided diagnosis schemes. This study proposes a new method for extracting nodular and linear patterns of various sizes in medical images. We have developed a morphology filter bank that creates multiresolution representations of an image. Analysis bank of this filter bank produces nodular and linear patterns at each resolution level. Synthesis bank can then be used to perfectly reconstruct the original image from these decomposed patterns. Our proposed method shows better performance based on a quantitative evaluation using a synthesized image compared with a conventional method based on a Hessian matrix, often used to enhance nodular and linear patterns. In addition, experiments show that our method can be applied to the followings: (1) microcalcifications of various sizes in mammograms can be extracted, (2) blood vessels of various sizes in retinal fundus images can be extracted, and (3) thoracic CT images can be reconstructed while removing normal vessels. Our proposed method is useful for extracting nodular and linear shadows or removing normal structures in medical images.

This paper proposes improved eigen-decomposition template matching. Eigen-decomposition template matching is a method of detecting the displacement and the rotational angle more accurately than traditional rotational correlation matching methods such as the cross-correlation method. The proposed method calculates the correlation by matching images determined by eigen-decomposition of the template image to the input image instead of matching the template image to the input image directly. This paper also proposes a method of detecting the displacement and rotational angle even if the input image includes large translation displacements and intensity variations. The experimental results show that the proposed method can detect the displacement and rotational angle more accurately than the previous method for images with noise and illumination variance. © 2012 Wiley Periodicals, Inc. Electron Comm Jpn, 95(10): 27–36, 2012; Published online in Wiley Online Library (wileyonlinelibrary.com). DOI 10.1002/ecj.11389

This paper proposes the improved eigen-decomposition template matching. The eigen-decomposition template matching is a method to detect the displacement and the rotational angle more accurately than the traditional rotational correlation matching such as the Cross-Correlation method. The proposed method calculates the correlation by matching the images which is determined by eigen decomposition of the template image to the input image instead of matching the template image to the input image directly. Furthermore, this paper proposes the method to detect the displacement and rotational angle even if the input image includes the large displacement translation and the intensity variations. The experimental results show that the proposed method can detect the displacement and rotational angle more accurately than the previous method for the image with a noise and an illumination variance.