Search

The developments in sensor technology, information processing, computer science, and artificial intelligence significantly improved robots’ autonomy. Robots’ external perception relies on sensing technology. Thus, capturing accurate sensor information is vital for ensuring robotic security and improving human-machine interaction performance. This paper classifies the main robotic sensors, describes multi-sensor information fusion and processing and contrasts the state-of-the-art sensor technologies for collaborative robots with other state-of-the-art technologies in related fields. In addition, this paper also introduces collaborative robots’ perception applications of the state-of-the-art representative products, the new designs for collaborative robots, the interactive applications of the intelligent Kinect sensor with collaborative robots, and the important applications of collaborative robots in the medical fields. Through a deep analysis of relevant information, this paper aims to introduce the integration of the state-of-the-art sensor technologies and collaborative robots, with hoping of guiding significance for the applications of robot sensors. This paper finally emphasizes the sensors’ impact on robot performance and discusses future research on sensor technologies in robotics.

COVID-19 pandemic has a catastrophic consequence globally since its first case was detected in December 2019, with an aggressive spread. Currently an exponential growth is expected. If not diagnosed at the proper time, COVID-19 may lead to death of the infected individuals. Thus, continuous screening, early diagnosis and prompt actions are crucial to control the spread and reduce the mortality. In this paper we focus on developing a Medical Diagnosis Humanoid (MDH) which is a cost effective, safety critical mobile robotic system that provides a complete diagnostic test to check whether an individual is infected by Covid-19 or not. This paper highlights the development of a system based on Artificial Intelligence for Medical Science, where humanoids can navigate through desired destinations, diagnose an individual for Covid-19 through various parameters and make a survey of a locality for the same. The humanoid uses the concept of real time data sensing and processing through machine learning produced by various sensors used in the context.

Aimed at improving the nonlinear integrated navigation solution performance of multiple low-cost sensors fusion, this paper presents a multilayer loosely-coupled, local-global, and step-optimized MF5DCKF (Multisensor Federated fifth-degree Cubature Kalman filter) state estimation algorithm for the small unmanned aerial vehicle (UAV). This method establishes a multilayer nonlinear integrated navigation model composed of the nonlinear attitude and heading reference system (AHRS) error model, strapdown inertial navigation system/global positioning system (SINS/GPS) error model, and strapdown inertial navigation system/barometer (SINS/BARO) error model to enhance the robustness and richness of the navigation module. Further, based on the above navigation models, a loosely-coupled error state fusion frame is designed to obtain the local convergent state vector. Simultaneously, a three-layer fifth-degree Cubature Kalman filter is proposed to improve the local state estimation accuracy. Subsequently, to optimize the estimated local state, this paper presents a novel distributed MF5DCKF scheme fusing the local state vector to calculate the global optimal state parameters in a step-optimized process. The experimental flight test results show that the proposed algorithm achieves a higher state solution accuracy and a better convergent performance compared with some conventional multisensor fusion algorithms. The new algorithm framework can provide applicability and reliability for the small UAV during the flight.

In robot visual servoing system, effective detected the region of interest area (ROI) in target images is the first problem should be solved, but the detection is susceptible to an unstructured environment. In this paper, an instance segmentation network algorithm based on Mask Region Convolutional Neural Networks (Mask R-CNN) framework is proposed for ROI image preprocessing. As instance segmentation technology can distinguish complex environmental information, so first with this advantage filter out message such as shape or similar area to target image, then with semantic segmentation technology add special category labels to the filtered images and distinguish different individual instances in similar categories. Finally through the series steps aforementioned, robot vision system can overcome the impact of environmental factors and identify the target image. The proposed method is used to detect target image under five constraints such as occlusion and reflection, result shows the algorithm can effectively deal with challenges that brought by complex constraints, and even can predict the location data of missing information based on some image information. In addition, based on the algorithm proposed in this paper, we used one seven axis robot visual servo platform, executed visual servoing experiment under different unstructured environments, further verifies the effectiveness of our proposed method.

Personalized mobile healthcare integrated with various wearable devices has become a significant area of interest in the present era. In the current research work, a flexible, wearable and disposable paper-based continuous skin temperature monitoring sensor for early medical prognosis and accurate diagnosis of body temperature-related ailments, such as COVID-19, is proposed. Conventional screen-printing and drop-casting techniques were used to fabricate the proposed sensor using MWCNTs as the sensing material and paper as the substrate. The linearity, stability, repeatability and durability of the sensors were tested from 29°C (room temperature) to 60°C. A thin sheet of PET was laminated over the sensor surface to ascertain its stability toward environmental effects and physical movements, and a response time of ~13 s and a recovery time of ~38 s with a sensitivity of -0.0685% °C-1 were recorded. The efficacy of the proposed sensor was ascertained by placing it at different body locations on a human subject and comparing it with a standard thermocouple and IR sensor. The sensor even helped to effectively distinguish minimal temperature variations between various regions of the body. Furthermore, the feasibility of the fabricated temperature sensor as a temperature-based tactile sensor for robotics/artificial skin applications and as a noncontact breath monitoring device for use in personalized healthcare monitoring applications was investigated.

Pedestrian positioning with wearable devices is a significant application of attitude tracking. It tracks the attitude (with heading angle being the most important part) of the device in real time and provides positioning services for users based on the information of step length provided by Pedestrian Dead-Reckon (PDR), which is a cheap and efficient positioning method at present. However, amid a train of positioning methods, the joint estimate of tracking is given by a train of methods based on the direction of gravity and the earths magnetic field direction. Considering the measurement of gravity that the gravity accelerometer is exposed to heavy noise due to the complex movement of human body during walking with uniform swing arm posture and forward speed, this paper proposed a novel estimate method based on the Kalman filter with multi-state constraints and the usage of low-cost sensors, which fulfills the estimation with the sequential observation of magnetic field. Compared with other related work, this method proposed in this paper eliminates the dependence on gravity direction, avoiding the influence of heavy noise caused by additional linear acceleration in motion state, and reduces the influence of insufficient observation when using magnetic field observation alone. The performance of the proposed method is evaluated by real-world experimentation results.

In this paper, we present a fast and robust visual-inertial odometry (VIO) algorithm, which realizes the tight coupling of monocular visual odometry (VO) and low-cost inertial measurement unit (IMU). Aiming at the problem of tracking loss caused by the interference of moving objects or blurred images in the traditional VIO algorithm, a sparse optical flow method combining edge detection algorithms is proposed. In the image preprocessing, Laplace edge detection is performed on the original image first, and an area with the most texture is searched according to the sharpened image. When tracking feature points with the sparse optical flow method, the feature points within the searched area are used as the main tracking targets to reduce the impact because of some unclear part in the image on the visual front-end. The trust region dogleg method is used for non-linear optimization in the back-end of the VIO system. The effectiveness of the proposed VIO system is validated on the public data set named EuRoC MAV and compared with the advanced VIO algorithm in recent years. Experimental results show that the new VIO system proposed in this paper has good robustness and can be applied to complex scenes such as fast motion, lighting changes, lack of features, and image blur. Compared with the traditional VIO algorithm, the pose estimation speed of the proposed VIO algorithm can be improved by 10% or more.

Cerebral palsy is one of the main factors leading to children’s disability. A large number of such children have hand motor dysfunction, such as limited range of motion, abnormal gestures, etc. Our goal is to design a prototype of wearable gesture training equipment for such children. For this purpose, this paper presents the development of a wireless smart glove to facilitate the accurate measurement of finger movement through the integration of multiple IMU sensors. Commercially available gaming gloves are not fitted with sufficient sensors for this particular application and most of them can only provide one degree of freedom for finger flexion and extension. In this paper, a data glove integrated with nine degree of freedom inertial sensors in conjunction with complex multi-sensor data fusion is developed. In our method, a single/dual state measurement update switching extended Kalman filter is proposed to estimate the spatial attitude of each finger segment. Through traversing the kinematic chain, a tree type hand dynamic model based on joint constraints is established to realize the real-time gesture tracking. Furthermore, a visual interface is developed to display the tracking effect of gestures, and the reliability of our algorithm is verified by optical contrast experiments, the repeatability of joint angle is also evaluated by kinematic analysis. In general, all the experimental results demonstrated that our proposed framework can accurately track the 3-D hand motion. This glove will help quantify joint stiffness and monitor patient progression during the rehabilitation training process.

With the increase of the world’s population, people’s demand for food is also increasing. However, due to various reasons, traditional agriculture cannot make full use of the land. Therefore, to ensure the food supply of the global population and promote the development of agriculture, the implementation of smart agriculture has become essential. The purpose of this research work is to study the smart agricultural wireless sensor routing protocols and node positioning algorithms based on the Internet of Things (IoT) technology. The present study first analyzes the classification of wireless sensor routing protocols. Based on the analysis and research of the Low Energy Adaptive Clustering Hierarchy protocol, the influence of factors such as node energy and distance are considered, and the network’s Low Energy Adaptive Clustering Hierarchy routing protocol is improved and extended its life. It introduced the classification of positioning algorithm, and then analyzed the DV-HOP positioning algorithm. Aiming at the problem of low positioning accuracy and large error of DV-HOP algorithm, an improved method of DV-HOP algorithm based on average HOP distance was proposed to make positioning more precise. Experimental results prove that the improved algorithm proposed in this paper has better performance. By testing the algorithm proposed in this paper, the improved DV-HOP algorithm reduces the positioning error by 30% compared with the original DV-HOP algorithm.

This paper aims to design and optimize a sports training system through wireless sensor network technology, which mainly includes a hardware system and software system. In the hardware system, the design of sensor nodes and the base station is realized, which can realize the real-time collection of movement parameters by the movement collectors. In the software system, the design of base station control, node control, and motion database software is realized, which can effectively collect, store, and analyze the motion parameters. Finally, the wireless sensor network-based sports training system is tested, and the test results show that the system designed in this paper can meet the needs of sports training use. The inertial measurement unit in the wearable device is used to collect the data generated by the human body during exercise, and low-power Bluetooth is used as the data transmission protocol between the wearable device and the smartphone. The server side provides computing resources in the form of cloud computing to achieve client-server interaction. In combination with the idea of service containerization, a deployment scheme for cloud services was proposed using container choreography, and compared to the pre-training period, the subjects’ trunk sway in the front and back directions decreased by 60.5% and 54.0%, and the trunk sway in the left and right directions decreased by 67.1% and 50.3% during and after training, respectively, and the ratio of time occupied by the equilibrium state increased by 63.5% and 47.4%, subjects’ balance was effectively improved.

Wireless Sensor Network (WSN) has wide range of applications including next generation intelligent IoT applications. Sensor nodes in WSNs don’t allow replacing batteries as phenomenon under consideration is not frequently accessible or inaccessible. Resource constrained WSN nodes run on batteries with limited capacity. For long running of the WSNs, energy consumption on each node and lifetime of overall network is a concern for IoT applications. To prolong lifetime of WSNs, use of energy efficient method is necessary and a challenging task. Most of the on-demand routing protocols use metrics like number of hops for the selection of path from the source node to the destination node. Purely hop count-based metrics leads to frequent broken paths as well as more energy consumption and results into reduced lifetime of IoT applications. Method proposed in this paper is a cross layer variant of AODV by replacing hop count metric with link quality and collision count. To establish a path, proposed method fetches link quality information from Physical layer and collision information from MAC layer to aid Network layer for making intelligent routing decisions using ZScore method. Using this information, proposed routing metric caters to select stable and sustainable routing path. Method proposed in this paper is implemented in well-known network simulator, NS-2 by incorporating necessary changes into existing physical and MAC layers as well as AODV protocol. A comparative analysis with existing methods is an indicator of improved performance in terms of energy efficiency, network lifetime, path stability and delay.

As an important part of the perception layer of the Internet of things, wireless sensor networks will provide sensing data for the application of the Internet of things, so it is necessary to optimize the routing path of wireless sensor nodes for the application of the Internet of things. Aiming at the problem of optimal selection of wireless sensor nodes for the Internet of things, this paper describes the mathematical model and abstracts it as a multi-objective optimization problem. Secondly, through the analysis of the deployment optimization process, this paper abstracts the optimization selection method of wireless sensor nodes facing the Internet of things and the guarantee method to avoid coverage holes. Then a node selection method based on a genetic algorithm is proposed to solve the problems of high redundancy and high energy consumption in the internet of things. In the application process of genetic algorithm, aiming at the possible problems of standard genetic algorithm, the similarity judgment is introduced into the crossover operation, and the operation of introducing new individuals is added into the genetic process. Finally, simulation experiments are carried out to verify the feasibility, efficiency, and parameter setting of the algorithm. The simulation results show that the proposed method can guarantee the coverage of the area to be monitored, reduce the network energy consumption and keep the energy consumption balanced.

The Internet of Drones (IoD) has become a hot research topic in academia, industry, and management in current years due to its wide potential applications, such as aerial photography, civilian, and military. This paper presents a comprehensive survey of IoD and its applications, deployments, and integration. We focused in this review on two main sides; IoD Applications include smart cities surveillance, cloud and fog frameworks, unmanned aerial vehicles, wireless sensor networks, networks, mobile computing, and business paradigms; integration of IoD includes privacy protection, security authentication, neural network, blockchain, and optimization based-method. A discussion highlights the hot research topics and problems to help researchers interested in this area in their future works. The keywords that have been used in this paper are Internet of Drones.

Identification and detection of traffic identifiers are some of the most challenging tasks in the field of autonomous driving. For object detection, how to obtain deep semantic information and shallow spatial information has always been a permanent subject. This paper proposes a dense network structure for the detection of traffic identifiers; the deep structure of the residual network is used to obtain the high-level semantic information of the detection target; the regression operation is used to obtain the underlying spatial information; the detection structure is higher than the previous The accuracy of the network structure and the generalization ability of the network has been improved; ASPP using the attention mechanism; the main function of ASPP is to obtain a continuous feature map; using the ASPP of the attention mechanism, the prior balance between the receptive field and the feature resolution map is solved; In the paper, a collection of residual networks is used to obtain a variety of acceptance domains; through a regression operation, an effective interactive connection between the receptive field and the acceptance domain is obtained; a deep network using regression residuals allows geometric transformation of the model during data training The ability to improve has been improved; at the same time, the use of a dense network structure has realized the diversity of different receptive fields in the process of multi-scale feature extraction.

An accurate and computationally efficient SLAM algorithm is vital for autonomous vehicles. Most modern SLAM systems use feature detection approaches to limit computational requirements. Feature detection through a 3D point cloud can be a computationally challenging task. In this paper, we propose a feature-based SLAM algorithm using 2D image projections of the 3D laser point cloud. We use a camera parameters matrix to rasterize the 3D point cloud to an image. Then ORB feature detector is applied to these images. The proposed method gives repeatable and stable features in a variety of environments. Based on such features, we can estimate the 6dof pose of the robot. For loop detection, we employ a 2-step approach, i.e., nearest key-frame detection and loop candidate verification by matching features extracted from rasterized LIDAR images. We evaluate the proposed system with implementation on the KITTI dataset. Through experimental results, we show that the algorithm presented in this paper can substantially reduce the computational cost of feature detection from the point cloud and the whole SLAM system while giving accurate results.

Aiming at the problems of low accuracy and low robustness in current prediction methods of rolling bearing performance degradation, this paper propose a performance degradation prediction method based on empirical mode decomposition (EMD) and support vector data description (SVDD). Unlike the features extracted by wavelet analysis, the EMD algorithm has better adaptive capability and can better avoid the interference of high frequency noise. fiber Bragg grating (FBG) are used to extract the bearing vibration signal. The use of FBG sensors can better avoid the interference of electromagnetic noise, so that the vibration signal can be accurately extracted in the industrial strong magnetic environment. The Empirical Mode Decomposition is used to decompose data and extract effective intrinsic mode function (IMF) components. Six features of Intrinsic Mode Function components are extracted to form feature vectors and used as training data. Because the IMF component obtained by EMD decomposition has removed most of the noise signals, the signal features are more significant and thus effectively improve the accuracy of subsequent recognition. This paper use Support Vector Data Description for performance degradation prediction and evaluates the degradation performance of the rolling bearing by calculating the degradation value. Experiments show the proposed method can detect the performance degradation process of the rolling bearing and effectively predict the remaining service life. Due to the high recognition accuracy, the proposed method is able to identify the current state of ball bearings and predict their remaining service life.

This paper presents multisensor image fusion of polarization and infrared imaging to detect defects in printed circuit boards (PCBs). Many existing automated optical inspection techniques rely on visible imaging sensors. However, collected images suffer from uneven brightness levels due to the influence of lighting environment, which may significantly affect detection accuracies. Polarization information characterizes material types, surface roughness, and geometric shape of an object. Thermal infrared imaging reveals heat radiation difference between the defect region and the background. Polarization and infrared imaging are not sensitive to background illumination and contrast. In this paper, we utilize polarization information as well as infrared imaging to detect the defects in PCBs that conventional optical inspection techniques cannot easily detect. We design a multi-source image acquisition system to simultaneously acquire brightness intensity, polarization, and infrared intensity. Then a Multisensor Lightweight Detection Network (MLDN), trained on the PCB dataset collected, fuses polarization information and the brightness intensities in the visible and thermal infrared spectra to detect defects in challenging lighting conditions. Experiment results show that the proposed network outperforms the state-of-the-art automated optical inspection techniques in terms of mean average precision.

There has been a significant increase in the number of total knee replacement (TKR) surgeries over the past few years, particularly among active young and elderly people suffering from knee pain. Continuous and optimal monitoring of the load on the knee is highly desirable for designing more reliable knee implants. This paper focuses on designing a smart knee implant consisting of a triboelectric energy harvester and a frontend electronic system to process the harvested signal for monitoring the knee load. The harvester produces an AC signal with peak voltages ranging from 10 V to 150 V at different values of knee cyclic loads. This paper demonstrates the measurement results of a PCB prototype of the frontend electronic system fabricated to verify the functionality and feasibility of the proposed approach for a small range of cycling load. The frontend electronic system consists of a voltage processing unit to attenuate high peak voltages, a rectifier and a regulator to convert the input AC signal into a stabilized DC signal. The DC voltage signal provides biasing for the delta-sigma analog-to-digital converter (ADC). Thus, the output of the triboelectric harvester acts as both the power signal that is rectified/regulated and data signal that is digitized. The power consumption of the proposed PCB design is approximately $5.35~\mu \text{W}$ . Next, the frontend sensor circuitry is improved to accommodate a wider range of cyclic load. These results demonstrate that triboelectric energy harvesting is a promising technique for self-monitoring the load inside knee implants.

In recent years, the Remaining Useful Life (RUL) prediction has become a hot spot in Prognostics and Health Management (PHM) research. High-accuracy RUL prediction can reduce the probability of accident occurrence and improve the reliability of the mechanical system. This paper proposes a novel two-channel hybrid model for RUL prediction based on a Capsule Neural Network and a Long Short-Term Memory Network (Cap-LSTM). Conventional Convolutional Neural Networks (CNN) are widely used in RUL prediction. However, the pooling layer of the conventional CNN only extracts the most active part of the multivariate time-series sensor data. Moreover, conventional CNN is not sensitive to the direction and spatial position of features and thus the feature information may not be fully used. To overcome these shortcomings, this paper uses the capsule neural network to directly extract the highly correlated spatial feature information, from the multivariate time-series sensor data, thus avoiding the loss of the spatial position relationship between local features and reducing the complexity of the model. In order to obtain the training and test samples, this paper uses the sliding time window method to preprocess the data. Meanwhile, the piece-wise linear function is used to represent the actual performance degradation of the engine. The NASA C-MAPSS dataset is used to verify the prediction effectiveness of the Cap-LSTM model. The state-of-the-art RUL prediction models are also introduced and compared to the proposed approach, providing results that show the superiority of the presented method.

Heart attack else wise termed as myocardial infarction (MI) causes irreparable death of cardiac muscles yielding the focal reason for most casualties among all cardiovascular diseases (CVDs’). A 12-lead electrocardiogram (ECG) generally depicts cardiac abnormalities and so customary deep learning (DL) approaches use the whole signal for binary detection purposes, that is separating healthy control (HC), and MI classes. This paper proposes an alternative approach where 21 temporal features in lieu of the temporal signal are collected from the 12 lead data to reduce redundancy and class imbalance keeping the vital information intact. Then these extracted features are fed into a detection model consisting of a one dimensional (1-D) convolutional neural network (CNN) and a bidirectional long short-term memory (bi-LSTM) layer which classifies into three classes, namely: HC, MI, and non-myocardial infarction (non-MI) subjects for a realistic and reliable assessment. The model’s performance is evaluated using 517 records acquired from the Physikalisch-Technische Bundesanstalt (PTB) database and a state-of-art overall accuracy of 99.246%, kappa of 0.983, and macro averaged F1 score of 98.86% were achieved using stratified 5-fold cross-validation. DL methods suffer to make unbiased decisions in the case of class imbalance due to an insufficient amount of data for a particular class and thus temporal features are employed to inherently reduce this problem. The successful performance of the extracted features depends on the precise detection of fiducial points, and so multiple novel algorithms have been introduced in this paper.

RFID-based pose perception can enable industrial automation applications such as industrial robot grasping. In this paper, a RFID pose estimation method based on classification algorithm and phase-position transformation model for moving objects is proposed, which converts the traditional pose estimation problem into a machine learning classification problem by dividing the direction angle value domain of the object into several classes. The phase information of multiple RFID tags attached to the object is transformed into position information using an unwrapped phase-position model, on which the input features of the classifier is constructed. A classifier based on the LightGBM framework is constructed and trained to realize the mapping between RFID phase information and the object’s pose. Extensive experiments demonstrate that the proposed method in this paper can accurately estimate the pose of moving objects in real time and successfully complete the robot grasping task for objects on the conveyor belt.

The paper proposes a data-driven pre-processing optimization for radar data using a parametric convolutional neural network. The proposed method is applied on human activity classification as a use case. Present radar-based activity recognition system exploit micro-Doppler signature by generating Doppler spectrograms or a temporal series of range-Doppler maps, followed by deep neural networks or machine learning approaches for classification. Those radar data representations are typically generated on the basis of short-time Fourier transformations. A Fourier transformation equally resolves the frequency space, which may be sub-optimal in some applications. Although deep convolutional neural networks (DCNN) have been shown to implicitly learn features from raw sensor data in other fields, such as speech recognition, yet, for the case of radar-based DCNNs, pre-processing is required to develop a scalable and robust classification or regression application. In this paper, we propose a parametric convolutional neural network that mimics the radar pre-processing across fast-time and slow-time radar data through 2D sinc filter or 2D wavelet filter kernels to extract features for classification of various human activities. During training only the filter parameters of the 2D sinc filters or 2D wavelets are learned, leading to optimized feature representation for the classification task. It is demonstrated that our proposed solution shows improved results compared to equivalent DCNN architectures that rely on Doppler spectrograms or radar data cubes as input data.

As a new development direction in the field of information science, wireless sensor network (WSN) has attracted wide attention of academia and industry with its huge application prospect. This research mainly discusses the automatic control system of balcony agriculture stereoscopic cultivation based on wireless sensor network. With the application background of environmental monitoring of protected cultivation, this paper analyzes the specific requirements of plant protected cultivation, and designs the overall scheme of wireless sensor network system with temperature, humidity and light as detection parameters. In this paper, a distance based routing algorithm is designed to realize the networking of wireless sensor networks, and the network simulation is carried out with TOSSIM. Aiming at the problem of packet loss in wireless network data transmission, this paper analyzes the operation mechanism and message structure of TinyOS operating system, and designs MAC protocol based on network synchronization to ensure reliable data transmission. The experimental results show that when the communication distance is 230 m, the two nodes can carry out normal data transmission; when the communication distance is less than 310 m, the bit error rate is very small; when the distance is more than 360 m, the bit error rate is significantly improved; when the distance is more than 400 m, the bit error rate is 100%. It can be seen that wireless sensor network can bring greater convenience to the acquisition of agricultural information, and it is of great significance to extend the application of wireless sensor network to the field of agricultural automation.

Leap Motion Controller (LMC) is a widely-used 3D user-interface device for virtual reality (VR) in hand tracking applications. However, the tracking area of a single LMC is not sufficient to cover the complete range of hand motion typically used in virtual reality applications, which can cause inconvenience and unnatural behavior of bare-hand interaction in a cooperated virtual environment. In this paper, we propose fusing the data from multiple LMCs to enlarge the tracking area. We present our shared-view calibration method based on a Least-squares Fitting algorithm. To track two hands in the enlarged tracking area, we propose a multi-targets tracking algorithm based on a Clustering-based Labeled Probability Hypothesis Density filter implemented by Gaussian mixture approach. A hand-recognition confidence is proposed to improve the tracking performance when hands are incorrectly recognized. The performance of the proposed algorithm was evaluated by three tests based on a five-LMCs system used on an Oculus Rift S. Results show that our system can track two hands stably in the range of 202.16 degrees horizontally and 164.43 degrees vertically, and the proposed algorithm shows superiority in tracking robustness under hand-recognition errors. The contribution of this paper is to provide a detailed guide for designing an enlarged hand-tracking system using sensor fusion.

The paper introduces a research program formulated to uncover and describe a complex nature of the sleep apnea disorders. This study include the physiological sensing and the signal processing oriented towards the mapping of a dynamical profile of physiological system represented by its complexity and variability. To reconstruct a heatmap of the dynamical features significant for triggering sleep disorders we collected a set of procedures dedicated to qualitative and quantitative depiction of the intra- and inter-events, and then adapted them to the use with a polysomnography data. Research protocol was organized with reference to the patients and modified PNEUMA model, and the COMPASS Toolbox devoted to time series exploration. The outcome novelty consists in the complementary characterization of the sleep apnea dynamics, measured at various levels of the system, but also the original statements on the sensitivity of fractal and network oriented algorithms applied to physiological data has been formulated in the report in reference to the temporal patterns encoded in polysomnography data, e.g. a detection of the central sleep apnea with the use of nasal airflow has been documented. The complementary approach proposed in the paper is a prerequisite to understand the SAS phenotyping, predict that modes and the SAS states, and formulate an efficient procedures for personalized patient care.

Contactless Human to Machine Interface (HMI) is an indispensable technology for handling the machine or equipment in a pandemic situation where the virus can spread through direct contact. The contactless hand gesture sensor is an essential part needed for the HMI system as previously mentioned. Several hand gestures with a similar Doppler response will become problems in applying Doppler radar as a hand gesture sensor that requires a more complex recognition method. This paper proposes spatial diversity implementation for obtaining more accurate hand gesture features to cope with this problem. Array radar was selected as a sensor configuration to create the spatial diversity feature. In this paper, an array configuration of four Continuous Wave (CW) radars is proposed as a contactless sensor for hand gestures. Peak detection based on cross-correlation was employed to determine the time position of the hand gesture Doppler response detected by each CW radar. The time position pattern then becomes a feature of each hand gesture used. The CW radar array is realized with an operating frequency of 10 GHz by using the HB 100 as a CW radar component. The experimental results show that the proposed method can distinguish the hand gesture feature with an accuracy of 96.6 % at a sensing distance of 50 cm. It can differentiate the hand gesture pairs that have the opposite direction movement with a similar Doppler effect, and also requires simple data processing for recognizing different hand gestures.

Pandemic situation such as COVID-19 boosted the demand for remote patient monitoring (RPM) system. The medical sensors attached to the human body in RPM system experience varying channel quality due to body movements. This paper analyzes the signal received from RPM sensors when a patient rotates by different angles while sitting on a chair as well as heed the use of a relay node placed on his/her body. Literature suggests many relay-based communication protocols to deliver physio-signals efficiently in an RPM application. However, limited studies have focused on the position of a relay node on the human body. In this paper, we empirically analyze the off-body communication path of sensor nodes by collecting data from different body orientations in a residential room. We estimate the path loss parameters for underweight, normal and overweight body mass index (BMI) categories. The estimated parameters are then used to simulate the physical layer of a home-based indoor RPM application. We inspect different relay node positions on the human body and allude an optimal position of the relay node that cover the transmission range of all sensors and provides an improved channel quality. We improve the Quality of Service (QoS) during non-line-of-sight (NLOS) situation and design an adaptive cross-layer communication protocol for WBANs.

Gait phase detection is essential for the wearable powered exoskeletons. This paper proposes an online gait phase detection method based on dynamic time warping mean (DTW-MEAN) templates using ground contact forces (GCFs). There are two important methods of gait phase detection, which are the template matching and statistical method. Template matching methods such as DTW are robust and have been widely used in the field of gait phase detection. However, the template matching is a kind of method based on distance measure, and the gait phase detection will ignore the coupling connections between the gait sequences; on the contrary, the statistical method can fuse these connections well. In this paper, a statistical mean method based on DTW is proposed to solve the problem of recognizing phases between gait sequences that are not correctly detected by single template methods. We tested our approach in complex environment (such as different terrains, different payloads, and different speeds) and gained over 95% accuracy and time difference below 25ms. Our proposed approach obtained better detection results with the advantage of no need to retrain templates.

Zero-velocity update method (ZUPT) is widely used in inertial measurement unit (IMU)-based pedestrian navigation systems for mitigating sensor drifting error. In the basic pedestrian dead reckoning (PDR) system, especially in the foot-tie PDR system, zero-velocity update method with Kalman filter are two core algorithms. In the basic PDR system, ZUPT usually uses a single threshold to judge the gait of pedestrians. A single threshold, however, makes ZUPT unable to accurately judge the gait of pedestrians in different road conditions. In this paper, we propose a new, redesigned ZUPT method without using additional equipment and filter algorithms to further improve the accuracy of correction results. The method uses a sliding detection algorithm to help re-detect the zero-velocity intervals, aiming to remove the pseudo-zero velocity interval and the pseudo-motion interval, as well as improving the performance of the ZUPT method. The method was implemented in a shoe-mounted IMU-based navigation system. For walking step detection tests, the accuracy of the proposed modified ZUPT method reached 87.24%, 25% higher than the conventional methods. In a long-distance walking path tracking test, the mean error of the estimated path of our method is 0.61 m, an 81.69% reduction compared to the conventional ZUPT methods. The details of the improved ZUPT method presented in this paper not only enable the tracking technology to better track a pedestrian’s step changes during walking, but also provide better calculation conditions for subsequent filter operations.

Human Activity Recognition (HAR) is a field that infers human activities from raw time-series signals acquired through embedded sensors of smartphones and wearable devices. It has gained much attraction in various smart home environments, especially to continuously monitor human behaviors in ambient assisted living to provide elderly care and rehabilitation. The system follows various operation modules such as data acquisition, pre-processing to eliminate noise and distortions, feature extraction, feature selection, and classification. Recently, various state-of-the-art techniques have proposed feature extraction and selection techniques classified using traditional Machine learning classifiers. However, most of the techniques use rustic feature extraction processes that are incapable of recognizing complex activities. With the emergence and advancement of high computational resources, Deep Learning techniques are widely used in various HAR systems to retrieve features and classification efficiently. Thus, this review paper focuses on providing profound concise of deep learning techniques used in smartphone and wearable sensor-based recognition systems. The proposed techniques are categorized into conventional and hybrid deep learning models described with its uniqueness, merits, and limitations. The paper also discusses various benchmark datasets used in existing techniques. Finally, the paper lists certain challenges and issues that require future research and improvements.

This paper presents a review of Tactile Sensing Systems, with an emphasis on their application to the non-invasive characterization of tumors. Emulating the perceptual mechanism of the human skin, the Tactile Sensing Systems characterize tumors using touch sensors by quantifying the mechanical properties such as size and elasticity. The authors survey several tactile transduction methods: capacitive, piezoresistive, piezoelectric, magnetic, and optical. The advantages and disadvantages of different tactile sensors are discussed. The complex human sense of touch is emulated using tactile sensor data, novel data processing algorithms, and near real-time interpretation in a human-readable format. Tactile Sensing Systems utilize tactile sensors and other subsystems to come up with accurate mechanical properties of the touched objects. We review Elasticity Determination Systems, which are a special case of Tactile Sensing Systems. These systems are based on capacitive sensors, piezoelectric sensors, elastography, and optical tactile sensors. Then the optical Tactile Sensing System is discussed in detail; architecture, sensing principle, and algorithms to compute a risk score. Moreover, a survey of multimodal Tactile Sensing Systems, which broaden the capabilities of existing tactile sensing systems, is presented. The paper concludes with discussions and future research directions.

Considering the adverse impact of speed measurement on the accuracy of pose estimation after a mobile robot slips, collides, or abducts, this paper proposes a monocular inertial simultaneous localization and mapping algorithm that includes wheel speed anomaly detection. The algorithm adds wheel speed measurement to the least squares problem in a tightly coupled manner and uses a nonlinear optimization method to maximize the posterior probability to solve the optimal state estimation. For the speed control of the Mecanum wheel, because the existing closed-loop speed control method cannot calculate the motion constraint error, this paper reports a design of a control method of the Mecanum wheel moving chassis based on torque control, which can use the motion constraint error to estimate the credibility of the wheel speed measurement to detect whether the chassis movement status is abnormal; meanwhile, to prevent the chassis speed measurement error from adversely affecting the robot pose estimation, this paper uses three methods to actively detect whether the chassis movement is abnormal, and analyze the chassis movement status in real time. When it is determined that the chassis has abnormal motion, the wheel odometer pre-integration measurement of the current frame is actively removed from the state estimation equation, thereby ensuring the accuracy of the pose estimation. Experimental results show the feasibility and effectiveness of the method proposed in this paper, and the algorithm is robust.

Images and videos in multimedia data are typical representation methods that include various types of information, such as color, shape, texture, pattern, and other characteristics. Besides, in video data, information such as object movement is included. Objects may move with time, and spatial features can change, which is incorporated in spatio-temporal relations. Many research studies have been carried out over time on information recognition by computers using low-level data in this connection. There is a semantic gap between low-level and high-level information in vocabulary representing human thinking. A substantial amount of research has been conducted on reducing the semantic gap, and it is focused on representation methods of logic. The goal of this study is to understand object movement and define spatio-temporal relations through mapping between vocabulary and the object movements. Ontology mapping is a method used to bridge the gap between low-level and high-level information. In this case, the spatio-temporal relation consists of temporal relations obedient to the passage of time, directional relations obedient to changes in object movement direction, changes in object size relations, topological relations obedient to changes in object movement position, and velocity relations using concept relations between topology models. In this paper, an ontology is used to define the inference rules using the proposed spatio-temporal relations and the use of Markov Logic Networks (MLNs) for probabilistic reasoning. Finally, the performed experiment and evaluation prove the verification recognition and understanding of object movements based on video data. This paper can be extended to retrieval and comparison between object movements, automatic annotation, and video summarization. The contributions of this paper include definition of the spatio-temporal relations of a region-based object, recognition of the semantic movements of moving objects, designing and constructing a spatio-temporal ontology, and Understanding the semantic movement of moving objects.

Currently healthcare data packets do not get any special priority while routing through the Internet of Things (IoT) networks. These data packets flow through routers using conventional QoS process which does not guarantee that a patient’s critical health data traveling in congested IoT network will actually be routed to doctors office on time. This leaves the remote medical treatment at risk with possible threat to patients’ lives. In this paper, we studied the current healthcare packet routing process in IoT and its performance issues in congested IoT networks, then proposed a sensor driven solution to prioritize healthcare data routing in congested IoT networks. In our proposed system, we add a healthcare data identifier in IP packet header at the sensor level, modify QoS software at the network router level, and provide the highest priority to healthcare data packet routing based on the healthcare data identifier seen at router QoS. A prototype has been built and tested by using TI Launchpads. Test data shows that healthcare packets with an identifier can route to doctors at 80% less latency than healthcare packets routed without an identifier. Additionally, the proposed system saves medical diagnostic cost, and more importantly, reduces the risk of losing human lives. A shorter version of this analysis has also been presented to IEEE conference. The detail analysis in this paper opens up multiple avenues for further research.

While the Internet of Things (IoT) has been instrumental in healthcare data transmission, it also presents vulnerabilities and security risks to patients’ personalized health information for remote medical treatment. Currently most published security solutions available for healthcare data are not focused on data flow all the way from IoT sensor devices placed on a patient’s body through network routers to doctor’s offices. In this paper, we studied how the IoT network facilitates healthcare data transmission for remote medical treatment, explored security risks associated with unsecured data transmission, especially between IoT sensor devices and network routers, and then proposed an encrypted security solution initiated at IoT sensor devices. Our proposed solution provides a cryptography algorithm embedded into the sensor device such that the packets generated with patient’s health data are encrypted right at the sensor device before being transmitted. The proof of concept has been verified using a lab setup with two level encryption at the IoT sensor level and two level decryption at the receiving end at the doctor’s office. Test results are promising for an end-to-end security solution of healthcare data transmission in IoT. This paper also opens up further research avenues on IoT sensor driven security.

The most daunting and conflicting challenges accompanying the wireless sensor networks are energy and security. And data aggregation and compression techniques are two of the effective ways to reduce energy consumption. As it is known that the radio transceiver consumes energy which is proportional to the number of bits transmitted on the network; hence sending fewer bits on the communication channel implies lesser energy consumption. This paper works on compressing the index of secure index on distributed data (SIDD) technique; to reduce the number of bits of an index that is transmitted on the communication channel. The objective being to reduce energy consumption of SIDD. In this paper, we have worked to reduce the number of bits of the index sent on the communication channel, deploying difference encoding. The compression mechanism has established an upper bound on the energy consumption whilst all data items were unique. The scheme is scalable and can be deployed for saving energy consumption.

3D optical sensors are becoming more and more popular in vision guidance of industrial robots and other industrial automation applications. Although research on high-speed 3D shape measurement (or imaging) has experienced tremendous growth over the past decades, simultaneously achieving high-speed and high-accuracy performance remains a major challenge in industrial practice. This paper presents a new FPGA architecture for the PMP algorithm and designs an high-speed embedded binocular structured light 3D measurement sensor. The sensor mainly contains a DLP projector, two cameras and a Xilinx Zynq-7000 UltraScale which make real-time computation possible. Experiments showed the time consuming of processing a set of ${9} \times {2}$ images with a resolution of ${2048} \times {1536}$ using the sensor proposed in this paper is 31.47 ms that has a better performance than 255ms on GPU. To the best of our knowledge, this is the first high-speed FPGA-based embedded binocular structured light 3D measurement sensor. This proposed 3D sensor can obtain ultra high quality 3D information in real time, it can be widely applied to robot random pin-picking and other industrial applications.

Mobile omnidirectional structured light vision is increasingly used in scene perception and robot navigation. A wide range of information is obtained by means of the vision system by only one image and laser image features are detected and extracted easily and quickly. In this paper a novel calibration method for mobile omnidirectional camera based on structured light is presented. Firstly, a set of parallel laser planes is emitted on the walls of corridor as auxiliary targets by structured light and intersects with wall orthogonally. Secondly, the constraint relationship is analyzed between the vanishing points in fisheye images and intrinsic parameters of imaging model. Finally, effects of the laser stripes’ interval and the angle between the wall which contains laser stripes and ground on calibration results are evaluated. Compared to Scaramuzza method, the calibration method shows its superiority in terms of both feasibility and efficiency. The method with the characteristic of self-calibration since the planar target is replaced by actively projected laser stripes. The result illustrates that our method has the advantages of simple and feasible operation, but result is effective and accurate. The calibration parameters are independent of the laser stripes’ interval and the angle between the wall and ground. Therefore, the method of the mobile omnidirectional structured light vision presented in this paper can be applied to many areas.

Through-wall mapping is an emerging field of research with promising applications varying from search and rescue, to health care and to security. Radar is a valuable tool in this process, mainly due to its long wavelength, which can pass through construction materials. However, this capability comes at a price, namely multipath reflections caused by the environment, which can reduce the usability of the produced map considerably. This paper proposes methods to detect and isolate these multipath reflections, eventually leaving out a usable map of the enclosed environment. For this purpose, two different radars, one operating at 24GHz and the other at 76GHz, were evaluated in various wall configurations constructed inside an enclosed space using portable wall segments. The testing arena was probed from the outside with the radars mounted on the top of a differential drive mobile robot. The paper shows that the proposed methods are effective in eliminating multipath reflections and in building an accurate representation of the environment using the radars.

The increasing bandwidth requirement of new wireless applications has lead to standardization of the millimeter wave spectrum for high-speed wireless communication. The millimeter wave spectrum is part of 5G and covers frequencies between 30 and 300 GHz corresponding to wavelengths ranging from 10 to 1 mm. Although millimeter wave is often considered as a communication medium, it has also proved to be an excellent 'sensor', thanks to its narrow beams, operation across a wide bandwidth, and interaction with atmospheric constituents. In this paper, which is to the best of our knowledge the first review that completely covers millimeter wave sensing application pipelines, we provide a comprehensive overview and analysis of different basic application pipeline building blocks, including hardware, algorithms, analytical models, and model evaluation techniques. The review also provides a taxonomy that highlights different millimeter wave sensing application domains. By performing a thorough analysis, complying with the systematic literature review methodology and reviewing 165 papers, we not only extend previous investigations focused only on communication aspects of the millimeter wave technology and using millimeter wave technology for active imaging, but also highlight scientific and technological challenges and trends, and provide a future perspective for applications of millimeter wave as a sensing technology., 36 pages, submitted to IEEE Sensors Journal

For in-line inspection of ferromagnetic materials, magnetic flux leakage (MFL) detection is one of the most common nondestructive testing methods. In practical applications of conventional MFL inspection, random mechanical vibration can cause the detection gesture of the probe, in particular, the lift-off and tilt detection angle, to continuously fluctuate throughout the inspection process, which introduces measurement errors to MFL signals. To address the problem and realize ultra-high-definition MFL detection, this paper proposed a new type of probe with dual diagonal distributed magnetic sensors for MFL detection. Based on the dual magnetic sensor probe structure and magnetic dipole model, this paper further developed analytical solutions of the real-time sensor lift-off, tilt detection angle, and defect depth. Additionally, this paper presented a compensation algorithm for MFL measurement errors caused by random lift-off and tilt detection angles. A practical implementation for the dual magnetic sensor probe in the actual MFL testing tool was given. The finite element simulation and physical experiments were designed to verify the feasibility of probe gesture compensation algorithm. Even under abnormal probe detection gesture case, MFL signals measured and compensated by the dual magnetic sensor probe were also in line with the engineering application, and measurement errors of MFL signals were reduced from more than 30 % to less than 10 %, which is conducive to the realization of ultra-high-definition MFL testing.

Electromagnetic wave is susceptible to the multipath effect, especially in complex indoor environment. As transmission of non-line of sight (NLOS), it leads to bigger measurement error, which makes positioning accuracy decline. In the paper, we aim to compensate the measurement error mainly affected by the NLOS to enhance the positioning accuracy. Conceptions of “virtual inertial point” and “environmental factor” are introduced, and moreover acceleration data of inertial measurement unit (IMU) are adopted. A novel and simple NLOS error compensation method for high accuracy indoor positioning system based UWB is proposed. Based on the principle of inertia, IMU can be used to fix motion trend of tag and to get coordinate of virtual inertial point. The measurement coordinate can be acquired by UWB system. We can calculate the distances between virtual inertial point and real measurement in same area, respectively under line of sight (LOS) and NLOS in advance. The ratio of two different distances can be defined as environmental factor, which is mainly used to reflect the influence of NLOS on positioning accuracy quantitatively relative to LOS in the paper. Environmental factor, as the weighted coefficient, is adopted to merge coordinates of the virtual inertial point and measurement to obtain the coordinate compensated under NLOS. It is a process of scaling measurement down to user-defined area for positioning accuracy requirement to reduce the impact of NLOS. At last, Kalman Filter is used to smooth the set of coordinate compensated under NLOS to further improve positioning accuracy. Experimental results show that our method can effectively compensate the NLOS error in the indoor positioning system based ultra-wide band (UWB) technology. The positioning accuracy is improved nearby 80% in NLOS area.

The path towards next-generation IoT is fundamentally characterized by artificial intelligence-based sensors. However, there are still some open issues, such as efficiency, accuracy, privacy, data trustworthiness, quality etc., that need to be addressed. Therefore, this special issue aims to solicit original papers to solve these issues. Finally, some excellent papers are selected in this special issue from about 160 submissions. Due to the limitation of the editorial space, only 3 papers will be introduced here.

Soft electronics have been widely applied to wearable electronics, hand gesture detection and human robot interaction (HRI). Soft wrist-worn sensor system (SWSS) with electromyography (EMG), strain/pressure sensing abilities has been created for continuous hand gesture recognition, which includes multi-source data sensing, data collection and process, and wireless communication. SWSS is conformal contact with skin surface as the softness of SWSS, and it would improve the wearability with the wrist during long-term hand gesture monitoring. Linear discriminant analysis (LDA) and support vector machines (SVM) algorithms are applied to hand gesture recognition with average accuracy 83.67%, 86.8% for Group #1 (wrist flexion (WF), wrist extension (WE), finger flexion (FF), finger extension (FE), radial deviation (RD), and ulnar deviation (UD)) and 84.71%, 88.53% for Group #2 (sign language 0-9 digits), respectively. It is found that the recognition accuracy has just a little decreased during the long-term hand gesture detection at different sessions (at initial time, one day after, three days after and seven days after). This paper demonstrated the feasibility of gesture recognition via SWSS on wrist, and it could be integrated into wrist-worn electronic system for human robot interaction.

Sound event detection (SED) and acoustic scene classification (ASC) are two key tasks related to each other in the field of computational auditory scene analysis. For example, during sound event detection, scene information can be used to exclude sound events that are unlikely to occur in this scene. In other words, scene information can improve the accuracy of sound event detection. However, existing works rarely detect sound events by considering acoustic scene information. Based on the internal relationship between sound events and scene information, this paper proposes a scene-dependent sound event detection (SDSED) approach, which combines scene information and sound event information using multi-task learning. In the proposed approach, we share common feature representation for the two tasks simultaneously. Meanwhile, a temporal attention mechanism is used to extract informative features from sound recordings. We test the proposed approach on Synthetic Sound Scenes dataset. Experimental results show that our proposed approach outperforms the state-of-the-art approaches. Compared with the referenced approach, our approach improves the segment-based F-score by 4.29% and reduces the segment-based error rate by 4.8%.

In the current industry world, the industrial robot has emerged as a critical device to make the manufacturing process more efficient through automation. However, abnormal operation of industrial robots caused by sensor failures may interrupt the entire manufacturing process, thereby increasing production costs. In this paper, we first propose a domain-specific framework consisting of offline training and online inference to effectively detect anomalies in the scenario of industrial robotic sensors. In a nutshell, the framework can be identified in three folds: (1) the offline training obtaining historical data from the database to train the time series and anomaly detection models; (2) the online inference deploying the offline trained models for online anomaly detection in real-time; (3) the incremental learning updating the online model for a new type of anomalies. We then propose an improved Generative Adversarial Networks (GANs) named MSGAN with the adaptive update strategy mechanism based on WGAN-GP to generate fake anomaly samples, improving anomaly detection accuracy. Specifically, the Wasserstein distance with the gradient penalty is introduced to improve training stability and sample quality. Moreover, adapting the generator’s complexity in robotic sensors, an adaptive update strategy based on the loss change ratio is adopted to speed up training convergence. Extensive experiments based on real robotic sensor datasets demonstrate the effectiveness of the proposed framework. Moreover, the results demonstrate that the detection accuracy can be improved by the synthetic samples based on the proposed MSGAN algorithm.

Non-destructive testing & evaluation techniques play an essential role in ensuring safety of materials in operation at various industry sectors. Pulse compressed favourable thermal wave imaging is one of the widely used non-destructive testing techniques due to its excellent noise rejection capabilities. However, the high dimensional thermal imaging data needs to be encoded into lossless compressed form to highlight the hidden defects inside the materials. This paper proposes a novel constrained and regularized autoencoder based thermography approach for sub-surface defect detection in a mild steel specimen. Certain properties such as non-correlation of encoded data, weight orthogonality, and weights with unit norm length have been highlighted which are non-existent in linear autoencoders but are responsible for better defect detection inside the materials inspected by frequency modulated thermal wave imaging. Novel constraints are formulated for autoencoder cost function to incorporate these significant properties. The proposed approach is able to provide better defect detection, in terms of signal to noise ratio of defects, than linear autoencoder as well as traditional principal component thermography approach. Also, non-correlation of encoded data is found to be the most significant factor in achieving better defect detection followed by properties ensuring weight orthogonality and weights with unit norm length.

The image-based virtual illumination technology directly changes the illumination effect of objects on the image. It does not require complex light propagation simulation calculations, it is an image-based rendering technology. The image mainly relies on the imaging of the visual sensor, and uses the virtual illumination technology of the image to illuminate the image obtained by the visual sensor. This is a new cross-cutting direction in computer vision, virtual reality and other fields. Face Illumination Swapping via Dense Landmark and Semantic Parsing is a major branch. Keeping the geometrical features of the target images and relighting the entire image instead of the face area are problems to be solved in the research. This paper based on the three-dimensional model to analyze the illumination information of the face images and re-render the illumination of the target face, and finally achieves the illumination swap between the two face images. We designed and implemented a 3DDFA-based face image illumination transfer method. First, 3DDFA is used to reconstruct the target face image. Estimate the surface normal and albedo. Then align and fill the surface normal and face parsing to illuminate the face image for light rendering and illumination transfer of the face images. Finally, the illumination analysis and re-rendering of face images based on 3DDFA are expanded to achieve the swap of illumination between face images. Experimental results show that this method can generate good effects of face image illumination transmission and swap while keeping the geometric features of the target images.

The proliferation of smartphones equipped with various sensors created a large user base, which includes robots (smartphone controlled) as well. The high processor enabled smartphones are used as software robots to track hour-to-hour activities of users using on-board motion sensors. The magnetometer, accelerometer, and gyroscope are non-permission based motion sensors. The logs of these sensors create discriminative patterns to identify users’ movements. This leverages an adversary to cause privacy threats using these non-permission based sensors without user consent. Recent research works also highlighted the privacy threats due to these motion sensors in smartphones. Therefore, in this paper, we present a potential misuse of technology where smartphones can be hacked by PIN Inference attack endangering the physical and software robots’ controlled user activities. We perform experiments to collect motion sensors’ data and infer 4-digit PIN of a smartphone. Our strategy is to detect every key pressed one by one constituting a 4-digit PIN. Our inference results are promising and prove that an adversary can make use of sensors’ data to infiltrate user’s sensitive information from smartphones. Our proposed model infers 84% of the PINs correctly within 40 attempts when tested on 50 PINs. Given real Android devices with different users, we can identify the PIN of smartphones by training our machine learning based model. We also compare our approach with existing state-of-the-art approaches to show the efficacy of our attack model.