Search

Object detection and tracking in camera images is a fundamental technology for computer vision and is used in various applications. In particular, object tracking using high-speed cameras is expected to be applied to real-time control in robotics. Therefore, it is required to increase tracking speed and detection accuracy. Currently, however, it is difficult to achieve both of those things simultaneously. In this paper, we propose a tracking method that combines multiple methods: correlation filter-based object tracking, deep learning-based object detection, and motion detection with background subtraction. The algorithms work in parallel and assist each other's processing to improve the overall performance of the system. We named it the "Mutual Assist tracker of feature Filters and Detectors (MAFiD method)". This method aims to achieve both high-speed tracking of moving objects and high detection accuracy. Experiments were conducted to verify the detection performance and processing speed by tracking a transparent capsule moving at high speed. The results show that the tracking speed was 618 frames per second (FPS) and the accuracy was 86% for Intersection over Union (IoU). The detection latency was 3.48 ms. These experimental scores are higher than those of conventional methods, indicating that the MAFiD method achieved fast object tracking while maintaining high detection performance. This proposal will contribute to the improvement of object-tracking technology. [ABSTRACT FROM AUTHOR]

This paper presents a high-speed skeletonization algorithm for detecting the skeletons of linear objects from their binary images. The primary objective of our research is to achieve rapid extraction of the skeletons from binary images while maintaining accuracy for high-speed cameras. The proposed algorithm uses edge supervision and a branch detector to efficiently search inside the object, avoiding unnecessary computation on irrelevant pixels outside the object. Additionally, our algorithm addresses the challenge of self-intersections in linear objects with a branch detection module, which detects existing intersections and initializes new searches on emerging branches when necessary. Experiments on various binary images, such as numbers, ropes, and iron wires, demonstrated the reliability, accuracy, and efficiency of our approach. We compared the performance of our method with existing skeletonization techniques, showing its superiority in terms of speed, especially for larger image sizes. [ABSTRACT FROM AUTHOR]

Making robots capable of pouring can be useful in both service and industrial applications. Considering the importance of controlling liquid vibration in mixing chemical reagents and other industrial applications, we investigated in the this study robotic pouring with the aim of controlling liquid vibration, more specifically, the beer-foam ratio during beer pouring. We propose a vision-based measurement method that can measure the liquid volume with an error of less than 5% in real time. Besides, together with a proposed robot pouring controller, we develop a robot pouring system that can control ratio of beer-foam volume with an error of less than 5% during pouring. The flexibility of the developed system was also demonstrated through experiments using different types of container and beer. [ABSTRACT FROM AUTHOR]

Multiple-target tracking across multiple camera views is required for various practical tracking and surveillance applications. This study proposes a seamless tracking method based on label transfer using high-speed image capture. The proposed method does not require camera calibration to estimate the extrinsic parameters, and can be applied with simple line calibration. Therefore, it is effective for large-scale camera networks. In addition, owing to the features of the high-speed image capture and self-window, our method ensures seamless tracking of objects across multiple camera views. The feasibility of the method is proven through an experiment using a high-speed camera network system. [ABSTRACT FROM AUTHOR]

This study focuses on robotic assistance for peg-and-hole alignment with micrometer-order clearance. The objective of the robotic assistance is to cooperate with a human operator based on a coarse-to-fine strategy in which the human operator conducts coarse alignment and the robotic assistance realizes fine alignment. Robotic-assisted fine alignment is achieved by mimicking the process toward annularity of an annular solar eclipse. The first principal axis of a specified image feature (we call it a eclipse feature) is calculated by subtracting the surfaces of a hole part (a small gear with an inner diameter of 1 mm) and a peg part (a shaft with a diameter of 0.95 mm). Accordingly, control strategy is developed to realize accurate alignment. Moreover, the effectiveness of the proposed method is verified by experimental evaluation. [ABSTRACT FROM AUTHOR]

In this study, we demonstrate the implementation of make-to-order bead art assembly without human intervention using dynamic compensation approach to achieve accurate real-time positioning and long-term adaptation for robotic automation in smart manufacturing. In the proposed framework, an industrial robot was designed to perform coarse global motion to implement low-bandwidth adaptation. Simultaneously, fine local motion to tackle real-time online uncertainties was achieved using an add-on robotic module to implement accurate positioning. The effectiveness of the proposed method was verified through experimental evaluations. [ABSTRACT FROM AUTHOR]

In multi-object tracking of identical objects, it is difficult to return to tracking after occlusions occur due to three-dimensional intersections between objects because the objects cannot be distinguished by their appearances. In this paper, we propose a method for multi-object tracking of identical objects using multiple high-speed vision systems. By using high-speed vision, we take advantage of the fact that tracking information, such as the position of each object in each camera and the presence or absence of occlusion, can be obtained with high time density. Furthermore, we perform multi-object tracking of identical objects by efficiently performing occlusion handling using geometric constraints satisfied by multiple high-speed vision systems; these can be used by appropriately positioning them with respect to the moving region of the object. Through experiments using table-tennis balls as identical objects, this study shows that stable multi-object tracking can be performed in real time, even when frequent occlusions occur. [ABSTRACT FROM AUTHOR]

Object tracking and pose estimation have always been challenging tasks in robotics, particularly for rotating objects. Rotating objects move quickly and with complex pose variations. In this study, we introduce a marker-based tracking and pose estimation method for rotating objects using a high-speed vision system. The method can obtain pose information at frequencies greater than 500 Hz, and can still estimate the pose when parts of the markers are lost during tracking. A robot catching experiment shows that the accuracy and frequency of this system are capable of high-speed tracking tasks. [ABSTRACT FROM AUTHOR]

Although they have been under development for years and are attracting a lot of attention, vision-based tactile sensors still have common defects—the use of such devices to infer the direction of external forces is poorly investigated, and the operating frequency is too low for them to be applied in practical scenarios. Moreover, discussion of the deformation of elastomers used in vision-based tactile sensors remains insufficient. This research focuses on analyzing the deformation of a thin elastic layer on a vision-based tactile sensor by establishing a simplified deformation model, which is cross-validated using the finite element method. Further, this model suggests a reduction in the number of markers required by a vision-based tactile sensor. In subsequent testing, a prototype HiVTac is fabricated, and it demonstrates superior accuracy to its vision-based tactile sensor counterparts in reconstructing an external force. The average error of inferring the direction of external force is 0.32 ∘ , and the root mean squared error of inferring the magnitude of the external force is 0.0098 N. The prototype was capable of working at a sampling rate of 100 Hz and a processing frequency of 1.3 kHz, even on a general PC, allowing for real-time reconstructions of not only the direction but also the magnitude of an external force. [ABSTRACT FROM AUTHOR]

This paper focuses on the teleoperation of a robot hand on the basis of finger position recognition and grasp type estimation. For the finger position recognition, we propose a new method that fuses machine learning and high-speed image-processing techniques. Furthermore, we propose a grasp type estimation method according to the results of the finger position recognition by using decision tree. We developed a teleoperation system with high speed and high responsiveness according to the results of the finger position recognition and grasp type estimation. By using the proposed method and system, we achieved teleoperation of a high-speed robot hand. In particular, we achieved teleoperated robot hand control beyond the speed of human hand motion. [ABSTRACT FROM AUTHOR]

Tracking and manipulating deformable linear objects (DLOs) has great potential in the industrial world. However, estimating the object's state is crucial and challenging, especially when dealing with heavy occlusion situations and physical properties of different objects. To address these problems, we introduce a novel tracking algorithm to observe and estimate the states of DLO. The proposed tracking algorithm is based on the Coherent Point Drift (CPD), which registers the observed point cloud, and the finite element method (FEM) model encodes physical properties. The Gaussian mixture model with CPD regularization generates constraints to deform a given FEM model into desired shapes. The FEM model encodes the local structure, the global topology, and the material property to better approximate the deformation process in the real world without using simulation software. A series of simulations and real data tracking experiments have been conducted on deformable objects, such as rope and iron wire, to demonstrate the robustness and accuracy of our method in the presence of occlusion. [ABSTRACT FROM AUTHOR]

In recent years, advances in CMOS imagers and AI have rapidly expanded the application fields of image processing. Spatial resolution, sensitivity, dynamic range, etc. for devices have dramatically improved, while various recognition functions have been implemented with the introduction of learning-based information processing, making progress day by day. However, the temporal resolution or temporal dynamics of an image has been limited to the video rate level of processing, because image processing has been required to realize the functions of the human eye. In the case of processing for high-speed moving objects or controlling machine dynamics as machine eyes, rather than processing in the range visible to the human eye, high-speed vision, i.e., high-speed image processing in a bandwidth that covers the dynamics of the object, and a system that utilizes such processing, are required. This special issue summarizes such advanced research on high-speed vision, highlighting its current status and future development in the areas of devices, systems, and application developments. High-speed vision has entered a new era, as the basic technology and various applications have been developed and new functions are being added one after another. We hope that this timely special issue will help its readers grasp this major technological trend and create new system values. [ABSTRACT FROM AUTHOR]

This paper presents a dynamic compensation concept to grapple with the dynamic defects of a traditional robot arm, especially while performing high-speed endpoint regulations. The proposed high-speed dynamic compensation concept offers a new point of view for cooperating with a traditional manipulator to realize highly dexterous performance of manipulations. The concept is realized through adoption of a high-speed light-weight actuator as well as endpoint closed loop configured high-speed cameras. The dynamic compensation is analyzed experimentally with 1000 Hz visual feedback and a high-speed finger for a robot arm in the case of one degree of freedom. The advantage of the proposed approach is that the modeling for the robot system’s dynamics is not needed, whereas it is necessary and trivial in order to realize high-speed regulations by traditional approaches. Thus, the control issue becomes easier with the proposed approach. As an application for this concept, fast peg-and-hole alignment with large position and attitude uncertainty is studied. The alignment algorithm is based on a visual compliance strategy. Alignment experiments show that with the proposed concept of dynamic compensation as well as visual compliant motion control, robust and fast convergence was realized for most cases. [ABSTRACT FROM AUTHOR]

A 75-year-old woman who had been taking methotrexate presented to our hospital for fever and dry cough. Chest computed tomography showed ground-glass opacity in the upper lobe of the right lung and foreign bodies in the lower lobe of the right bronchus. During bronchoscopy, foreign bodies were removed from the airway. We found increased levels of lymphocytes and a high CD4/CD8 ratio in the bronchoalveolar lavage fluid. Transbronchial lung biopsy revealed lymphocytic infiltration. Methotrexate was discontinued, and the imaging findings improved. Methotrexate-induced lung injury does occur unilaterally. Foreign bodies in the airway might also trigger unilateral methotrexate-induced lung injury. [ABSTRACT FROM AUTHOR]

In a previous study, we developed a force-detecting interface to operate an electric wheelchair for persons with muscular dystrophy. However, the force input method of the interface tends to be directly influenced with respect to disturbance caused by road surface conditions. We proposed a disturbance removal method to detect users' intentions using experimental results on the roads, where shocks or vibrations are experienced. Moreover, because of the structure of the electric wheelchair, the vehicle behaviour may easily become unstable due to disturbances such as the caster behaviour and the shape of the road (bumps and slopes). When an user's intended input differs from the vehicle's behaviour (brought about by these disturbances), the user has the burden of revising the command to the wheelchair. We proposed the longitudinal stability control system for the wheelchair utilising an in-wheel-motor system to improve the vehicle stability with respect to environmental disturbance. [ABSTRACT FROM AUTHOR]

This paper focuses on development of a high-speed, low-latency telemanipulated robot hand system, evaluation of the system, and demonstration of the system. The characteristics of the developed system are the followings: non-contact, high-speed 3D visual sensing of the human hand, intuitive motion mapping between human hands and robot hands, and low-latency, fast responsiveness to human hand motion. Such a high-speed, low-latency telemanipulated robot hand system can be considered to be more effective from the viewpoint of usability. The developed system consists of a high-speed vision system, a high-speed robot hand, and a real-time controller. For the developed system, we propose new methods of 3D sensing, mapping between the human hand and the robot hand, and the robot hand control. We evaluated the performance (latency and responsiveness) of the developed system. As a result, the latency of the developed system is so small that humans cannot recognize the latency. In addition, we conducted experiments of opening/closing motion, object grasping, and moving object grasping as demonstrations. Finally, we confirmed the validity and effectiveness of the developed system and proposed method. [ABSTRACT FROM AUTHOR]

In this research, we focused on Human-Robot collaboration. There were two goals: (1) to develop and evaluate a real-time Human-Robot collaborative system, and (2) to achieve concrete tasks such as collaborative peg-in-hole using the developed system. We proposed an algorithm for visual sensing and robot hand control to perform collaborative motion, and we analyzed the stability of the collaborative system and a so-called collaborative error caused by image processing and latency. We achieved collaborative motion using this developed system and evaluated the collaborative error on the basis of the analysis results. Moreover, we aimed to realize a collaborative peg-in-hole task that required a system with high speed and high accuracy. To achieve this goal, we analyzed the conditions required for performing the collaborative peg-in-hole task from the viewpoints of geometric, force and posture conditions. Finally, in this work, we show the experimental results and data of the collaborative peg-in-hole task, and we examine the effectiveness of our collaborative system. [ABSTRACT FROM AUTHOR]

In this paper, we develop a new sensor network system with a high sampling rate (over 500 Hz) based on the simultaneous synchronization of clock and data acquisition for integrating the data obtained from various sensors. Hence, we also propose a method for the synchronization of clock and data acquisition in the sensor network system. In the proposed scheme, multiple sensor nodes including PCs are connected via Ethernet for data communication and for clock synchronization. The timing of the data acquisition of each sensor is locally controlled based on the PC’s clock locally provided in the node, and the clocks are globally synchronized over the network. We construct three types of high-speed sensor network systems using the proposed method: the first one is composed of a high-speed tactile sensor node and a high-speed vision node; the second one is composed of a high-speed tactile sensor node and three acceleration sensor nodes; and the last one is composed of a high-speed tactile sensor node, two acceleration sensor nodes, and a gyro sensor node. Through experiments, we verify that the timing error between the sensor nodes for data acquisition is less than 15 μs, which is significantly smaller than the time interval of 2 ms or a shorter sampling time (less than 2 ms). We also confirm the effectiveness of the proposed method and it is expected that the system can be applied to various applications. [ABSTRACT FROM AUTHOR]

A 78-year-old man with a history of surgical resection for ameloblastoma 31 years earlier visited our hospital for prolonged cough. Chest computed tomography showed multiple nodules in both lungs. Although there was no local recurrence in the mandible, the specimen taken from a transbronchoscopic bronchial biopsy showed recurrent ameloblastoma. Despite receiving no treatment, the disease in our patient remained clinically stable for 8.4 years. Chest physicians should be aware that pulmonary malignant ameloblastoma can first relapse several decades after curative surgery. In addition, pulmonary malignant ameloblastoma without local recurrence may be associated with a good prognosis.

We propose a 500-frames-per-second high-speed vision (HSV) sensor network that acquires frames at a timing that is precisely synchronized across the network. Multiple vision sensor nodes, individually comprising a camera and a PC, are connected via Ethernet for data transmission and for clock synchronization. A network of synchronized HSV sensors provides a significantly expanded field-of-view compared with that of each individual HSV sensor. In the proposed system, the shutter of each camera is controlled based on the clock of the PC locally provided inside the node, and the shutters are globally synchronized using the Precision Time Protocol (PTP) over the network. A theoretical analysis and experiment results indicate that the shutter trigger skew among the nodes is a few tens of microseconds at most, which is significantly smaller than the frame interval of 1000-fps-class high-speed cameras. Experimental results obtained with the proposed system comprising four nodes demonstrated the ability to capture the propagation of a small displacement along a large-scale structure.

It is traditionally difficult to implement fast and accurate position regulation on an industrial robot in the presence of uncertainties. The uncertain factors can be attributed either to the industrial robot itself (e.g., a mismatch of dynamics, mechanical defects such as backlash, etc.) or to the external environment (e.g., calibration errors, misalignment or perturbations of a workpiece, etc.). This paper proposes a systematic approach to implement high-performance position regulation under uncertainties on a general industrial robot (referred to as the main robot) with minimal or no manual teaching. The method is based on a coarse-to-fine strategy that involves configuring an add-on module for the main robot's end effector. The add-on module consists of a 1000 Hz vision sensor and a high-speed actuator to compensate for accumulated uncertainties. The main robot only focuses on fast and coarse motion, with its trajectories automatically planned by image information from a static low-cost camera. Fast and accurate peg-and-hole alignment in one dimension was implemented as an application scenario by using a commercial parallel-link robot and an add-on compensation module with one degree of freedom (DoF). Experimental results yielded an almost 100% success rate for fast peg-in-hole manipulation (with regulation accuracy at about 0.1 mm) when the workpiece was randomly placed.

In this paper, a new measurement method of the rotation angle of a rotor, which is named the 'visual encoder,' was proposed. This method is based on the principles of the vision-based method and the optical encoder, and realized by using a high-speed vision system. The visual encoder shows advantageous features such as non-contact, high-resolution and robustness against the free motion and the fluctuation of the rotation axis. A high resolution method to increase the measurement resolution was also suggested. The accuracy and the robustness of the visual encoder were confirmed through the experimental verifications, and the operation was possible at 6,000 rpm even under the fluctuation of rotation axis.

Nilotinib, a second-generation tyrosine kinase inhibitor (TKI), has been approved for first-line chronic myeloid leukemia (CML) treatment. The improved clinical response of nilotinib over that of the first generation TKI, imatinib, has been thought to be a result of its high potency of inhibition of BCR-ABL kinase. This study aimed to characterize differences between nilotinib and imatinib in the intracellular accumulation and cytotoxic effect on the CML cell line K562. Accumulation of nilotinib in K562 cells was from 4.7- to 9.0-fold higher than that of imatinib. The cytotoxic effect of nilotinib on K562 cells was 14.2-fold higher than that of imatinib. Inhibition experiments in K562 cells, and examination of the cellular uptake using influx transporter-transfected human embryonic kidney (HEK) 293 cells, suggested that the influx transporters OCT1 and OATP1A2, which have been reported to mediate accumulation of imatinib in CML cells, contributed little to the uptake of nilotinib. Nilotinib was found to accumulate in imatinib-resistant K562 (K562/IM) cells overexpressing the efflux transporter P-glycoprotein (P-gp), although cytotoxic assays showed that K562/IM cells displayed 20000-fold greater resistance to nilotinib over the parent K562 cells. In conclusion, the present findings suggest that intracellular accumulation of nilotinib in CML cells contributes to its clinical response and efficacy in CML patients. Although nilotinib has been reported to be effective against imatinib-resistant ABL kinase mutants, the drug could not overcome imatinib resistance acquired by P-gp-overexpression. These results imply that classification of mechanisms of drug resistance is important for suitable strategies to treat imatinib-resistant CML patients.

A precise and convenient high-performance liquid chromatography (HPLC) method has been established to assay nilotinib in human plasma. Chromatographic separation of nilotinib was performed on a LiChrosphere(®)100 RP-18(e) column (250 mm×4.0 mm, 5 µm) using a mixture of acetonitrile and 0.01 M phosphate buffer (pH 3.0) (42 : 58, v/v) under isocratic conditions at a flow rate of 1.0 ml/min with ultraviolet (UV) detection at 266 nm. The calibration curve showed linearity at concentrations between 250 ng/ml and 5000 ng/ml (r(2)>0.999). The mean±S.D. absolute recovery of nilotinib from plasma was 99.2±3.3%. The coefficients of variation of both intra- and inter-day precision were below 9.1%. These results indicate that this new HPLC-based quantification may be useful for therapeutic drug monitoring of nilotinib to help manage treatment in patients with chronic myeloid leukemia in clinical practice.