Search

We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including stereo cylindrical 360$^\circ$ RGB video at 15 fps, 3D point clouds from two Velodyne 16 Lidars, line 3D point clouds from two Sick Lidars, audio signal, RGB-D video at 30 fps, 360$^\circ$ spherical image from a fisheye camera and encoder values from the robot's wheels. Our dataset incorporates data from traditionally underrepresented scenes such as indoor environments and pedestrian areas, all from the ego-perspective of the robot, both stationary and navigating. The dataset has been annotated with over 2.3 million bounding boxes spread over 5 individual cameras and 1.8 million associated 3D cuboids around all people in the scenes totaling over 3500 time consistent trajectories. Together with our dataset and the annotations, we launch a benchmark and metrics for 2D and 3D person detection and tracking. With this dataset, which we plan on extending with further types of annotation in the future, we hope to provide a new source of data and a test-bench for research in the areas of egocentric robot vision, autonomous navigation, and all perceptual tasks around social robotics in human environments.

The consensus problem is relevant to different areas ranging from biology, social psychology, and physics to power systems and robotics. Two crucial aspects of the design of a consensus system are the implementation issues that arise in densely connected networks and the presence of malicious agents that try to cause a deviation from a synchronization state. In this article, we introduce a formulation to design the topology of a consensus network to improve its resilience to attacks while remaining sparse and consistent with the a priori structural relations between the agents. Through mathematical analysis and simulations on artificial and real-world cases, we show the benefits and usefulness of using this strategy to design resilient and structurally sparse consensus networks.

This paper presents a framework for telepresence operation by touching live video on a touchscreen. Our goal is to enable users to use a smartpad to teleoperate everyday objects by touching the objects live video they are watching. To this end, we coined the term teleinteractive device to describe such an object with an identity, an actuator, and a communication network. We developed a touchable live video image-based user interface (TIUI) that empowers users to teleoperate any teleinteractive device by touching its live video with touchscreen gestures. The TIUI contains four modules touch, control, recognition, and knowledge to perform live video understanding, communication, and control for telepresence operation. We implemented a telepresence operation system that consists of a telepresence robot and teleinteractive devices at a local site, a smartpad with the TIUI at a remote site, and communication networks connecting the two sites. We demonstrated potential applications of the system in remotely controlling telepresence robots, opening doors with access control panels, and pushing power wheelchairs. We conducted user studies to show the effectiveness of the proposed framework.

Summary Surgeons and their patients recognise that one of the major advances in surgical technique over the last 20 years has been the growth of minimal access surgery by means of laparoscopic and robotic approaches. Partnerships with industry have facilitated the development of advanced technical instruments, light sources, recording devices and optics which are almost out of date by the time they are introduced to surgical practice. However, lest we think that technological innovation is entirely a modern concept, we should remember that our predecessors were masters of their craft and able to apply new technologies to surgical practice. The history of minimal access surgery can be traced back to approximately 5000 years ago and this review aims to remind us of the achievements of historical doctors and engineers, as well as bring more modern developments to wider attention. This review will comprise a three-part series: Part I 3000BC to 1850 Early instruments for viewing body cavities Part II 1850 to 1990 Technological developments Part III 1990 to present Organisational issues and the rise of the robots

Soft robots can provide advantages for medical interventions given their low cost and their ability to change shape and safely apply forces to tissue. This article explores the potential for their use for endoscopically-guided balloon dilation procedures in the airways. A scalable robot design based on balloon catheter technology is proposed, which is composed of five balloons together with a tip-mounted camera and LED. Its design parameters are optimized with respect to the clinical requirements associated with balloon dilation procedures in the trachea and bronchi. Possessing a lumen to allow for respiration and powered by the pressure and vacuum sources found in a clinical procedure room, the robot is teleoperated through the airways using a game controller and real-time video from the tip-mounted camera. The robot design includes proximal and distal bracing balloons that expand radially to produce traction forces. The distal bracing balloon is also used to perform balloon dilation. Three actuation balloons, located between the bracing balloons, produce elongation and bending of the robot body to enable locomotion and turning. An analysis of the actuation balloons, which incorporate helical coils to prevent radial collapse, provides design formulas by relating geometric parameters to such performance criteria as maximum change in actuator length and maximum robot bending angle. Experimental evaluation of a prototype robot inside rigid plastic tubes and

Thomson coil actuators consist of an electric coil repelling a metallic disk. They are considered the fastest electromagnetic actuators for short travels and for this reason are favored for DC circuit breakers. Although the basic physics are understood, the actuator’s few components are very interdependent, making for a difficult analysis. Actuator design so far has thus been limited to trial and error based on finite-element models. In this paper, a novel approach is presented based on the electromagnetic frequency response of the actuator. This method is shown to be an intriguing analysis tool and various actuator design directions are presented. Experimental results provide a confirmation of the analysis.

In recent era, IoT-assisted robotics (IoRT) plays a major part in collaborating IoT sensor nodes, robots and human beings to establish a collaborative communication. In this paper, we develop an IoRT system with a modern architecture that encompasses the integration of IoT sensor nodes and robots in smart cities. The study develops a cloud platform to access smart vehicle mobility data in smart cities without collision. A framework is then developed to assist the robots and IoT sensor node in managing and making quick decisions on traffic collision. Hence, it is developed to connect, share, and exchange the information between the cloud and the IoT nodes/robots. The experimental validation shows improved traffic control and collision free traffic in smart cities than other smart city traffic management mechanism.

Multigoal reinforcement learning (RL) extends the typical RL with goal-conditional value functions and policies. One efficient multigoal RL algorithm is the hindsight experience replay (HER). By treating a hindsight goal from failed experiences as the original goal, HER enables the agent to receive rewards frequently. However, a key assumption of HER is that the hindsight goals do not change the likelihood of the sampled transitions and trajectories used in training, which is not the fact according to our analysis. More specifically, we show that using hindsight goals changes such a likelihood and results in a biased learning objective for multigoal RL. We analyze the hindsight bias due to this use of hindsight goals and propose the bias-corrected HER (BHER), an efficient algorithm that corrects the hindsight bias in training. We further show that BHER outperforms several state-of-the-art multigoal RL approaches in challenging robotics tasks.

Background Inguinal lymph node dissection (ILND) is an essential step in both treatment and staging of several malignancies including penile and vulvar cancers. Various open, video endoscopic, and robotic-assisted techniques have been utilized so far. In this review, we aim to describe available minimally invasive surgical approaches for ILND, and review their outcomes and complications. Methods The PubMed, Wiley Online Library, and Science Direct databases were reviewed in February 2020 to find relevant studies published in English within 2000–2020. Findings There are different minimally invasive platforms available to accomplish dissection of inguinal nodes without jeopardizing oncological results while minimizing postoperative complications. Video Endoscopic Inguinal Lymphadenectomy and Robotic Video Endoscopic Inguinal Lymphadenectomy are safe and achieve the same nodal yield, a surrogate metric for oncological adequacy. When compared to open technique, Video Endoscopic Inguinal Lymphadenectomy and Robotic Video Endoscopic Inguinal Lymphadenectomy may offer faster postoperative recovery and fewer postoperative complications including wound dehiscence, necrosis, and infection. The relatively high rate and severity of postoperative complications hinders utilization of recommended ILND for oncologic indications. Minimally invasive approaches, using laparoscopic or robotic-assisted platforms, show some promise in reducing the morbidity of this procedure while achieving adequate short and intermediate term oncological outcomes.

A 69-year-old man was referred to the hepatobiliary surgeons for mild enlargement of an asymptomatic cystic liver lesion found on routine screening in 2017 that measured 3.7×3.6×4.3 cm. Work-up with MRI revealed a complex multilocular cyst that had enlarged to 6.6×5.5×4.6 cm. Other work-up was unremarkable. He had a Eastern Cooperative Oncology Group score of 1; therefore, a surgical excision was planned due to the possibility for malignancy. A robotic approach with enucleation of the lesion was undertaken, with plans for return for a wider resection if pathological examination revealed malignancy. The lesion was noted to be a biliary adenofibroma, an exceptionally rare lesion that is thought to be benign, but requires excision due to potential malignant degeneration. The patient was discharged home the following day and has had minimal pain in his postoperative course.

This paper discusses the digital automation workflows and co-design methods that made possible the comprehensive robotic prefabrication of the BUGA Wood Pavilion - a large-scale production case study of robotic timber construction. Latest research in architectural robotics often focuses on the advancement of singular aspects of integrated digital fabrication and computational design techniques. Few researchers discuss how a multitude of different robotic processes can come together into seamless, collaborative robotic fabrication workflows and how a high level of interaction within larger teams of computational design and robotic fabrication experts can be achieved. It will be increasingly important to discuss suitable methods for the management of robotics and computational design in construction for the successful implementation of robotic fabrication systems in the context of the industry. We present here how a co-design approach enabled the organization of computational design decisions in reciprocal feedback with the fabrication planning, simulation and robotic code generation. We demonstrate how this approach can implement direct and curated reciprocal feedback between all planning domains - paving the way for fast-paced integrative project development. Furthermore, we discuss how the modularization of computational routines simplify the management and computational control of complex robotic construction efforts on a per-project basis and open the door for the flexible reutilization of developed digital technologies across projects and building systems., European Regional Development Fund, Deutsche Forschungsgemeinschaft, Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg, Projekt DEAL, Bundesgartenschau Heilbronn 2019 GmbH, GETTYLAB

Hand detection is a crucial technology for space human-robot interaction (SHRI), and the awareness of hand identities is particularly critical. However, most advanced works have three limitations: 1) the low detection accuracy of small-size objects; 2) insufficient temporal feature modeling between frames in videos; and 3) the inability of real-time detection. In the article, a temporal detector (called TA-RSSD) is proposed based on the SSD and spatiotemporal long short-term memory (ST-LSTM) for real-time detection in SHRI applications. Next, based on the online tubelet analysis, a real-time identity-awareness module is designed for multiple hand object identification. Several notable properties are described as follows: 1) the hybrid structure of the Resnet-101 and the SSD improves the detection accuracy of small objects; 2) three-level feature pyramidal structure retains rich semantic information without losing detailed information; 3) a group of the redesigned temporal attentional LSTM (TA-LSTM) is utilized for three-level feature map modeling, which effectively achieves background suppression and scale suppression; 4) low-level attention maps are used to eliminate in-class similarity between hand objects, which improves the accuracy of identity awareness; and 5) a novel association training scheme enhances the temporal coherence between frames. The proposed model is evaluated on the SHRI-VID dataset (collected according to the task requirements), the AU-AIR dataset, and the ImageNet-VID benchmark. Extensive ablation studies and comparisons on detection and identity-awareness capacities show the superiority of the proposed model. Finally, a set of actual testing is conducted on a space robot, and the results show that the proposed model achieves a real-time speed and high accuracy.

A major challenge for online and data-driven model learning in robotics is the high sample-complexity. This hinders its efficiency and practical feasibility for lifelong learning, in particular for developmental robots that autonomously bootstrap their sensorimotor skills in an open-ended environment. In this work, we propose new methods to mediate this problem in order to permit the learning of robot models online, from scratch, and in a learning while behaving fashion. Exploration is utilized and autonomously driven by a novel intrinsic motivation signal which combines knowledge-based and competence-based elements and surpasses other state-of-art methods. In addition, we propose an Episodic Online Mental Replay to accelerate the online learning, to ensure the sample-efficiency, and to update the model online rapidly. The efficiency as well as the applicability of our methods are demonstrated with a physical 7-DoF Baxter manipulator. We show that our learning schemes are able to drastically reduce the sample complexity and learn the data-driven model online, even within a limited time frame.

Background Robot-assisted laparoscopic prostatectomy (RALP) is a favored surgical approach for treating prostate cancer. However, RALP does not decrease postoperative pain significantly despite its minimal invasiveness. The pain associated with robot-assisted surgery is most severe during the immediate postoperative period. We aimed to demonstrate that preoperative rectus sheath block (RSB) can reduce acute pain after RALP. Methods A prospective non-randomized study with two parallel groups was performed from June 2020 to August 2020. A total of 100 patients undergoing RALP were divided into two groups: the RSB group (n = 50) and the non-RSB group (n = 50). Ultrasound-guided RSB was performed preoperatively only in the RSB group. The primary outcome of the study was the visual analog scale (VAS) pain score during coughing (VAS-C) 1 h after surgery. In addition, the VAS pain score at rest (VAS-R) and the VAS-C were assessed up to 24 h after surgery. The doses of postoperative opioids consumed were also recorded. Results The RSB group had a significantly lower VAS-C 1 h after RALP (58 [47–73] vs. 74 [63–83] mm, p = 0.001). In addition, the RSB group had significantly lower VAS-R and VAS-C scores, and postoperative opioid requirement, up to 6 h after surgery compared to the non-RSB group. Moreover, the VAS-R was significantly lower in the RSB group than in the non-RSB group 24 h after surgery. Conclusion Preoperative RSB significantly improved analgesia during the early period after RALP. The long-term analgesic efficacy of RSB needs further study.

This article presents a versatile soft robotic gripper system whereby its fingers can be reconfigured into different poses such as scoop, pinch, and claw. This allows the gripper to efficiently and safely handle food samples of different shapes, sizes and stiffness such as uncooked tofu and broccoli floret. The 3D-printed fingers were tested to last up to 25 000 cycles without significant changes in the curvature profile and force output profile. A benchmark experiment was conducted to evaluate the performance of the gripper and state-of-the-art gripping solutions. Capability of versatile soft gripper was optimized by integrating vision and tactile sensing facilities. An object recognition system was developed to identify food samples such as potato, broccoli, and sausage. Position and orientation of food samples were identified and pick-and-place pathway was optimized to achieve the best gripping performance. Flexible tactile sensors were integrated into soft fingers and closed-loop force feedback control system was developed. This allowed the gripper to automatically explore and select the most stable grip pose for different food samples. Integration of vision and force feedback system ensure that objects detected by the system would be firmly gripped. The reconfigurable soft robotic gripper system has been demonstrated to perform high-speed pick-and-place tasks (∼3 s per item) with object recognition system, making it a potential solution to food and grocery supply chain needs.

Mobile robots are expected to move safely and efficiently in socially-compliant ways in dense crowds. In this paper, a unified multiple-motion-mode framework is proposed to achieve socially-compliant navigation in dense crowds. The proposed framework consists of three successive phases, the identification of pedestrian groups and prediction of their trajectories, the generation of candidate local trajectories of the robot, and the determination of an optimal local trajectory. The pedestrians are grouped according to distances among pedestrians and their velocities and a predictor predicts the groups' trajectories. Three robot motion modes, moving-solo, pedestrian-following, and courteous-stopping, are proposed based on a systematic analysis and classification of dense crowds and they are utilized to generate candidate local trajectories. A composite metric is proposed to determine the best local trajectory by simultaneously considering the robot motion safety, efficiency, and stability of the trajectory. The proposed robot navigation framework is evaluated both in simulations and real-world experiments with various scenarios. The results show that the proposed framework surpasses the state-of-the-art methods in terms of efficiency and social compliance.

This study aims to explore the usability of the virtual robotics programming curriculum (VRP-C) for robotics programming teaching. Pre-service computer science (CS) teachers were trained for robotics programming teaching by using VRP-C in a scientific education activity. After training, views of the participants were revealed by using a scale and an evaluation form consisting of open-ended questions. Results show that VRP-C is compatible with the curriculum for robotics programming teaching in schools, and pre-service CS teachers tend to use VRP-C in their courses. They think that VRP-C will be beneficial for robotics programming teaching in terms of content, functionality, and cost. Compatibility, visual design, feedback, time management, fiction, gamification, and cost are the characteristics that increase the usability of VRP-C. VRP-C can be used as an online tool for robotics programming training due to the necessity of transition to distance education because of the COVID-19 pandemic.

The developments in sensor technology, information processing, computer science, and artificial intelligence significantly improved robots’ autonomy. Robots’ external perception relies on sensing technology. Thus, capturing accurate sensor information is vital for ensuring robotic security and improving human-machine interaction performance. This paper classifies the main robotic sensors, describes multi-sensor information fusion and processing and contrasts the state-of-the-art sensor technologies for collaborative robots with other state-of-the-art technologies in related fields. In addition, this paper also introduces collaborative robots’ perception applications of the state-of-the-art representative products, the new designs for collaborative robots, the interactive applications of the intelligent Kinect sensor with collaborative robots, and the important applications of collaborative robots in the medical fields. Through a deep analysis of relevant information, this paper aims to introduce the integration of the state-of-the-art sensor technologies and collaborative robots, with hoping of guiding significance for the applications of robot sensors. This paper finally emphasizes the sensors’ impact on robot performance and discusses future research on sensor technologies in robotics.

Aimed at improving the nonlinear integrated navigation solution performance of multiple low-cost sensors fusion, this paper presents a multilayer loosely-coupled, local-global, and step-optimized MF5DCKF (Multisensor Federated fifth-degree Cubature Kalman filter) state estimation algorithm for the small unmanned aerial vehicle (UAV). This method establishes a multilayer nonlinear integrated navigation model composed of the nonlinear attitude and heading reference system (AHRS) error model, strapdown inertial navigation system/global positioning system (SINS/GPS) error model, and strapdown inertial navigation system/barometer (SINS/BARO) error model to enhance the robustness and richness of the navigation module. Further, based on the above navigation models, a loosely-coupled error state fusion frame is designed to obtain the local convergent state vector. Simultaneously, a three-layer fifth-degree Cubature Kalman filter is proposed to improve the local state estimation accuracy. Subsequently, to optimize the estimated local state, this paper presents a novel distributed MF5DCKF scheme fusing the local state vector to calculate the global optimal state parameters in a step-optimized process. The experimental flight test results show that the proposed algorithm achieves a higher state solution accuracy and a better convergent performance compared with some conventional multisensor fusion algorithms. The new algorithm framework can provide applicability and reliability for the small UAV during the flight.

Robots are a combination of mechatronics, computer science, and artificial intelligence. Robotics is a branch of engineering that involves the conception, design, manufacture, and operation of actions. Whenever a robot has to interact with the human-society, it has to adopt a special skill called Human-Robot Interaction and thus the term Social-Robot comes into account. A social robot has to be able to express emotions, communicate with high-level dialogue, use natural cues, and learn to recognize models of other agents. An autonomous social robot cannot follow orders instead of doing something on its own. To make the robot more interactive and communicative, lots of sensors and modules have to be used along with its moving mechanism. Therefore, a social robot becomes complex and expensive. To overcome the issue of the complexity and costliness, in this paper, a design of social robot using a combination of embedded systems, the Internet of Robotic Things (IORT) and Android operating system has been introduced to be interactive and communicative to human, be intelligent enough to solve complex mathematics and be able to follow the operator’s command simultaneously. By using the Internet as the robot’s source of information and the Android phone as the robot’s sensory and control system partially, and adding them all to the robot’s embedded system wirelessly, we have not only become able to make the robot more advanced and intelligent, but also reduce the cost of construction by a significant amount.

BACKGROUND Chronic urinary retention (CUR) is a frequent complication after orthotopic neobladder (ONB) reconstruction in women. To decrease CUR, several open surgical modifications to provide back support to the ONB have been established on the basis of pelvic anatomical differences between females and males. OBJECTIVE To illustrate our technique for robotic intracorporeal reconfiguration of ONB as integrated into our open surgical approach to provide back support to the ONB with round ligaments in women. DESIGN, SETTING, AND PARTICIPANTS From November 2017 to April 2021, 28 patients underwent robotic intracorporeal ONB with a minimum of 6 mo of follow-up at a single centre. SURGICAL PROCEDURE We performed robotic radical cystectomy, pelvic lymphadenectomy, and a complete intracorporeal ONB suspended with round ligaments (rONB). Our surgical procedure is demonstrated in the accompanying video. MEASUREMENTS Demographics and clinical and pathological data were collected. Perioperative and 90-d complications and 6-mo functional outcomes were compared for the rONB group (n = 12) and the patients receiving a traditional ONB (tONB; n = 16). RESULTS AND LIMITATIONS The median total operative time was 305 min (interquartile range [IQR] 270-370) for tONB and 303 min (IQR 287-330) for rONB. The median estimated blood loss was 325 ml (IQR 200-700) for tONB and 350 ml (IQR 262-600) for rONB. Some 50% of the tONB group and 41.7% of the rONB group experienced low-grade complications. A total of 12.5% tONB and 8.3% rONB patients experienced high-grade complications with neobladder-vaginal fistula. The cumulative risk of CUR was 37.5% in the tONB group and 16.7% in the rONB group. This study is limited by the small sample size and the short follow-up period. CONCLUSIONS We established a feasible surgical technique for a robotic intracorporeal ONB configuration suspended with round ligaments. This may prevent the occurrence of emptying dysfunction in women. PATIENT SUMMARY We describe our stepwise technique for creating a new bladder within the body that is suspended with round ligaments. Patients undergoing removal of the bladder for bladder cancer may benefit from this technique in terms of better urinary function and the advantages of a robotic surgical approach.

Monocular 3D object detection is a low cost and challenging task for autonomous vehicles and robotics. Utilizing a monocular image for 3D object detection is served as an auxiliary module for autonomous vehicles and is a growing concern recently. Currently, the expensive lidar and stereo cameras have a predominant performance on accurate 3D object detection, whereas monocular based methods are considerably lower in performance. This performance gap is minimized by reforming the monocular based method as a single internal network. We exploit the correlation between 2D and 3D detection spaces, enabling 3D boxes to leverage feature maps generated in image space. The 2D and 3D proposals are extracted through a proposal generation network that is enhanced and utilized for estimating accurate 3D detection and localization. Experimental results on the KITTI dataset demonstrate that in comparison to other monocular object detection methods the proposed method considerably improved the accuracy of 3D object detection. The mean average precision of 3D object detection in front view is improved to 25% and the bird's eye view to 32% for the car class on the moderate difficulty level.

The ability of robots to write Chinese strokes, which is recognized as a sophisticated task, involves complicated kinematic control algorithms. The conventional approaches for robotic writing of Chinese strokes often suffer from limited font generation methods, which limits the ability of robots to perform high-quality writing. This paper instead proposes a developmental evolutionary learning framework that enables a robot to learn to write fundamental Chinese strokes. The framework first considers the learning process of robotic writing as an evolutionary easy-to-difficult procedure. Then, a developmental learning mechanism called “Lift-constraint, act and saturate” that stems from developmental robotics is used to determine how the robot learns tasks ranging from simple to difficult by building on the learning results from the easy tasks. The developmental constraints, which include altitude adjustments, number of mutation points, and stroke trajectory points, determine the learning complexity of robot writing. The developmental algorithm divides the evolutionary procedure into three developmental learning stages. In each stage, the stroke trajectory points gradually increase, while the number of mutation points and adjustment altitudes gradually decrease, allowing the learning difficulties involved in these three stages to be categorized as easy, medium, and difficult. Our robot starts with an easy learning task and then gradually progresses to the medium and difficult tasks. Under various developmental constraint setups in each stage, the robot applies an evolutionary algorithm to handle the basic shapes of the Chinese strokes and eventually acquires the ability to write with good quality. The experimental results demonstrate that the proposed framework allows a calligraphic robot to gradually learn to write five fundamental Chinese strokes and also reveal a developmental pattern similar to that of humans. Compared to an evolutionary algorithm without the developmental mechanism, the proposed framework achieves good writing quality more rapidly.

Background Focal therapy (FT) for prostate cancer is less invasive than radical treatment but carries a risk of recurrence. Salvage robot-assisted radical prostatectomy (S-RARP) is a possible option after FT failure. Objective To evaluate the impact of FT on functional and oncological outcomes following S-RARP. Design, setting, and participants In a retrospective analysis of data from a prospectively collected institutional database, 53 patients who underwent S-RARP following failure of focal ablation were selected as group I; patients who had whole-gland ablation and external beam therapy were excluded. This group was matched to a control sample (matched at ratios of 1:1, 1:2, 1:3, 1:4) of men who had undergone primary RARP, using age, prostate-specific antigen (PSA), PSA density, body mass index, Sexual Health Inventory for Men score, American Urological Association symptom score, Charlson comorbidity index, prostate weight, preoperative Gleason score (GS), and history of smoking as variables. Surgical procedure S-RARP after FT was performed using a standardized technique developed at our institute with the da Vinci Xi Surgical System. Measurements Oncological and functional outcomes were compared between the S-RARP and primary RARP groups. Results and limitations There was no difference in estimated blood loss (p = 0.8) between the 1:1 matched groups, but operating room time was significantly longer for S-RARP (p = 0.007). The primary RARP group had a higher proportion of patients who underwent a full nerve-sparing procedure. The S-RARP group had higher incidence of positive surgical margins (40% vs 15%; p = 0.008), GS ≥8 (25% vs 15%; p = 0.07), and positive lymph node status (9.4% vs 5.7%; p = 0.02). There was no significant difference in overall complications between the groups. The primary RARP group had a higher incidence of lymphocele drainage after surgery (15% vs 0%; p = 0.006). The main limitation of the study is its retrospective design. Conclusions S-RALP after FT failure is feasible; however, surgery following FT leads to poorer oncological and functional outcomes. Despite the targeted nature of FT, significant nonfocal collateral damage is evident in tissues surrounding the prostate, which in turn translates to poorer functional outcomes after S-RARP. Patient summary We studied the surgical challenges during robot-assisted removal of the prostate after previous focal treatment (FT) for prostate cancer and compared the outcomes to those for robot-assisted prostate removal in patients who had no previous FT. We found that this technique is safe and effective with a limited risk of complications, but poor urinary and sexual functional outcomes.

Mammals have ability to perform accurate and robust path integration-based metric navigation even in the absence of visual or other environmental cues, which provides a new idea to find brain-inspired solutions to tackle the problems of serious drift of IMU-based inertial navigation of UAV when the external sensory cues are not available. Multi-scale grid cells in the medial entorhinal cortex are thought to be a fundamental portion of mammals’ ability to perform 3D path integration-based metric navigation. This paper studies and presents, for the first time, a neural system to implement path integration-based metric navigation in 3D environments integrating networks of encoding and decoding multi-scale grid cells using neural dynamic models, i.e. three-dimensional continuous attractor network and neural cliques, respectively. Experimental results show the neural system can successfully path integrate self-motion information for large-scale 3D navigation and provides robust and error-correcting position information, displaying possible neural solution to overcome serious drift of IMU-based inertial navigation of UAV in the absence of external sensory cues.

Learning to adapt one’s gait with environmental changes plays an essential role in locomotion of legged robots which remains challenging for constrained computing resources and energy budget, as in the case of edge-robots. Recent advances in bio-inspired vision with dynamic vision sensors (DVS) and associated neuromorphic processing can provide promising solutions for end-to-end sensing, cognition and control tasks. However, such bio-mimetic closed-loop robotic systems based on event-based visual sensing and actuation in the form of spiking neural networks (SNN) have not been well explored. In this work, we program the weights of a bio-mimetic multi-gait central pattern generator (CPG) and couple it with DVS-based visual data processing to show a spike-only closed-loop robotic system for a prey-tracking scenario. We first propose a supervised learning rule based on stochastic weight updates to produce a multi-gait producing Spiking-CPG (SCPG) for hexapod robot locomotion. We then actuate the SCPG to seamlessly transition between the gaits for a nearest prey tracking task by incorporating SNN based visual processing for input event-data generated by the DVS. This for the first time, demonstrates the natural coupling of event data flow from event-camera through SNN and neuromorphic locomotion. Thus, we exploit bio-mimetic dynamics and energy advantages of spike-based processing for autonomous edge-robotics.

Space robotics has been used extensively in complex space missions. Rigid-manipulator space robots may suffer from rigid-body collisions with targets. This collision is likely to cause damage to the space robot and the target. To overcome such a problem, a novel Continuum-Manipulator Space Robot (CMSR) for performing on-orbit servicing missions is proposed in this paper. Compared with rigid-manipulator space robots, CMSRs are able to perform compliant operations and avoid rigid-body collisions with a target. The CMSR consists of two kinds of flexible components, including solar arrays and continuum manipulators. The elastic vibrations of these flexible components disturb the position and attitude of CMSRs. The beating phenomenon introduced by the energy transfer among these flexible components can cause damage to solar arrays. The complicated dynamic coupling poses enormous challenges in dynamic modeling and vibration analysis. The dynamic model for CMSRs is derived and the mechanism of the beating phenomenon is analyzed in this paper. Simulation results show that an obvious beating phenomenon occurs and the amplitude of the solar arrays increases significantly when the natural frequencies of two kinds of flexible components are close. A method is provided to avoid the beating phenomenon.

Personalized mobile healthcare integrated with various wearable devices has become a significant area of interest in the present era. In the current research work, a flexible, wearable and disposable paper-based continuous skin temperature monitoring sensor for early medical prognosis and accurate diagnosis of body temperature-related ailments, such as COVID-19, is proposed. Conventional screen-printing and drop-casting techniques were used to fabricate the proposed sensor using MWCNTs as the sensing material and paper as the substrate. The linearity, stability, repeatability and durability of the sensors were tested from 29°C (room temperature) to 60°C. A thin sheet of PET was laminated over the sensor surface to ascertain its stability toward environmental effects and physical movements, and a response time of ~13 s and a recovery time of ~38 s with a sensitivity of -0.0685% °C-1 were recorded. The efficacy of the proposed sensor was ascertained by placing it at different body locations on a human subject and comparing it with a standard thermocouple and IR sensor. The sensor even helped to effectively distinguish minimal temperature variations between various regions of the body. Furthermore, the feasibility of the fabricated temperature sensor as a temperature-based tactile sensor for robotics/artificial skin applications and as a noncontact breath monitoring device for use in personalized healthcare monitoring applications was investigated.

Farms and mines facing a shortage of workers willing to do dangerous or low-paying jobs are turning to cutting-edge robotics, cloud computing, and artificial intelligence (AI) to leapfrog ahead of ...

It is important to accurately obtain the motion information of the ego vehicle and surrounding vehicles for autonomous vehicles to plan safe trajectories in complicated traffic environments. In this paper, a LiDAR-based estimation method is developed to simultaneously identify the pose and the velocity information of an ego vehicle and its surrounding moving obstacles. Specifically, a pose estimation network is designed to estimate the poses of both the ego vehicle and the obstacles only with the continuous point clouds obtained by the LiDAR mounted on the ego vehicle. In the network, PointNet++ is utilized as the backbone to extract point-wise features and divide the points into the static part and the moving part. The former is used to estimate the ego vehicle's pose, while the latter is applied for the obstacle pose identification. Then, a reduced-order observer is designed to estimate the velocities, whose convergence is proved with the Lyapunov theory. Finally, both simulation and experiment results are provided to show the effectiveness of the proposed method.

The problem of calibrating extrinsic parameter between camera and inertial measurement unit (IMU) using an industrial robotic manipulator has been studied. This generates a result of hand-eye/robot-world/camera-IMU calibration in a simultaneous fashion. The developed method is free of inertial integration over time and thus is robust to uncertain IMU biases. It is derived that the problem can be solved via a simultaneous optimization of hand-eye/robot-world/camera-IMU transformations. The resulted optimization is highly non-convex on the special Euclidean group and we give globally optimal solutions. Experiments verify that the proposed method is capable of estimating accurate calibration parameters. Comparative studies between representatives show the global optimality of the proposed method. The new simultaneous method is capable of conducting calibration of a robot/camera/IMU combination. The designed method guarantees the global optimality thus the accuracy is ensured. The developed globally optimal solutions will also be computationally efcient on modern industrial computers. Finally, we show that the proposed method can give accurate calibration results for a stereo/IMU sensor combination.

We address the motion planning problem for a robotic system whose configuration manifold contains a group of rotations. Our approach is applied to a free-floating space robot composed of a three-dimensional base (a spacecraft) and an anthropomorphic onboard manipulator. The robot is actuated by the torques exerted at the joints of the onboard manipulator. A coordinate-free representation of rotations is utilized. The Lagrangian formalism is employed in order to derive a dynamics model of the robot that takes the form of a control system defined on the group of rotations and the joint space of the onboard manipulator. Using the conservation of angular momentum of the robot, a Jacobian motion planning algorithm is designed relying on the Endogenous Configuration Space Approach. The performance of the algorithm is verified by computer simulations.

Tunable lens technology inspired by the human eye has opened a new paradigm of smart optical devices for a variety of applications due to unique characteristics such as lightweight, low cost, and facile fabrication over conventional lens assemblies. The fast-growing demands for tunable optical lenses in consumer electronics, medical diagnostics, and optical communications require the lens to have a large focal length modulation range and high compactness. Herein, for the first time, an all-solid tunable soft lens driven by highly transparent dielectric elastomer actuators (DEAs) based on poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate) and waterborne polyurethane (PEDOT:PSS/WPU) transparent electrodes is developed. The deformation of the tunable soft lens is achieved by the actuation of DEAs, mimicking the change of the surface profile of the human eye to achieve remarkable focal length variations. Upon electrical activation, this tunable soft lens can vary its original focal length by 209%, which is one of the highest among current tunable soft lenses and far beyond that of the human eye. This study demonstrates that transparent DEAs are capable of achieving focus-variation functions, and potentially useful in artificial robotic vision, visual prostheses, and adjustable glasses, which will induce significant effects on the future development of tunable optics.

The automotive industry has witnessed an increasing level of development in the past decades; from manufacturing manually operated vehicles to manufacturing vehicles with a high level of automation. With the recent developments in Artificial Intelligence (AI), automotive companies now employ blackbox AI models to enable vehicles to perceive their environments and make driving decisions with little or no input from a human. With the hope to deploy autonomous vehicles (AV) on a commercial scale, the acceptance of AV by society becomes paramount and may largely depend on their degree of transparency, trustworthiness, and compliance with regulations. The assessment of the compliance of AVs to these acceptance requirements can be facilitated through the provision of explanations for AVs' behaviour. Explainability is therefore seen as an important requirement for AVs. AVs should be able to explain what they have 'seen', done, and might do in environments in which they operate. In this paper, we provide a comprehensive survey of the existing body of work around explainable autonomous driving. First, we open with a motivation for explanations by highlighting and emphasising the importance of transparency, accountability, and trust in AVs; and examining existing regulations and standards related to AVs. Second, we identify and categorise the different stakeholders involved in the development, use, and regulation of AVs and elicit their explanation requirements for AV. Third, we provide a rigorous review of previous work on explanations for the different AV operations (i.e., perception, localisation, planning, control, and system management). Finally, we identify pertinent challenges and provide recommendations, such as a conceptual framework for AV explainability. This survey aims to provide the fundamental knowledge required of researchers who are interested in explainability in AVs., Comment: Accepted for publication in IEEE Transactions on Intelligent Transportation Systems

Research on soft artificial muscles (SAMs) is rapidly growing, both in developing new actuation ideas and improving existing structures with multifunctionality. The human body has more than 600 muscles that drive organs and joints to achieve desired functions. Inspired by the human muscles, this article presents a new type of SAM fiber formed from twisting and braiding soft hydraulic filament artificial muscles with high aspect ratio, high strain, and high energy efficiency. We systematically investigated the relationship between input pressure and output elongation as well as contraction force of the new muscles using different configurations in terms of an array of single and multiple muscles arranged in nontwisting (or straight), twisting, and braiding variants. Experimental results revealed that the twisting and braiding configurations greatly enhanced the muscle elongation and generated force compared with their nontwisting/braiding counterparts. To demonstrate the new muscles' usability, we implemented several muscle variants to bidirectionally manipulate 3D-printed human fingers and elbow, mimicking the human upper limb with a full range of motion. We also created a bioinspired growing soft tubular muscle that could simultaneously exert longitudinal and radial expansion upon pressurization, similar to that of auxetic metamaterial structures. The new growing soft tubular muscles were experimentally validated and the results showed that they could be potentially implemented in several emerging applications, including smart compression garments, stent-like supporting devices, and tubular grippers for medical use.

Actively reconfigurable flexible electronics enabled by robotic technologies would provide opportunities to extensively broaden the application areas of flexible electronics compared with their passively flexible counterparts. Diverse reconfigurable shapes can be enabled by localized control of the film-type electronics while keeping a thin form factor for flexible electronics. The flexible electronics are usually designed to be thin to lower mechanical stress or strain when bending. In this article, we present a locally actuatable film designed to control actuating regions, bending directions, and curvatures by the electrical inputs. The film is in a thin form factor with multichannels realized by the proposed method of crimping electrical wires over shape memory alloy wires. The industry standard process should be convenient in terms of scaling up, increasing channels, or adding reconfigurable shapes for new applications. We demonstrate shape-changeable displays, self-rollable photovoltaics, and light art to prove the feasibility of the concept.

This paper proposes a new reconfigurable intelligent surface based three-dimensional beam tracking method to solve the beam tracking problems for the unmanned aerial vehicle with obstacles in communication channels. The proposed beam tracking method can not only regulate the reconfigurable intelligent surfaces to achieve the beam tracking of the obstructed communications, but also optimize the transmission efficiency of the communication. Firstly, a reconfigurable intelligent surface is proposed, which can correlate the transmission signals by adjusting the phase-shift matrix. Meanwhile, a new communication channel is constructed according to the reconfigurable intelligent surface, which consists of two parts. The first one is the channel between the unmanned aerial vehicle and the reconfigurable intelligent surface, the other is the channel between the reconfigurable intelligent surface and the ground base station. Note that the transmission performance of the communications can be optimized by adjusting the phase shift of each uniform linear array on the reconfigurable intelligent surface. Then, a new beam tracking method for the unmanned aerial vehicle with obstructed communications is proposed on the basis of the reconfigurable intelligent surface. By proposing the mixed genetic algorithm, the estimation accuracy of the azimuth and elevation angles is improved to enhance the performance of the beam tracking. Finally, the simulations are provided to verify the effectiveness of the proposed three-dimensional beam tracking method with reconfigurable intelligent surface.

In robotics, decoding complex human movements of multiple degrees of freedom (DOFs) from surface electromyography (sEMG) remains challenging. Recently, the rapid development of artificial intelligence (AI) technology provides a new solution to this problem. In this paper, we propose an AI-based framework that consists of a series of deep learning approaches, for achieving a precise and robust decoding on three-dimensional (3D) wrist movements. A previously developed device (WMD) was utilized to tag the myoelectric signals with 3D kinematic labels. A public dataset (HIT-SimCo) was established, wherein the sEMG signals were collected from diverse wrist movements and multiple subjects. A lightweight convolutional neural network (CNN) was constructed, which can extract the motion-related features directly from raw sEMG, and make motion predictions in an end-to-end manner. In addition, several data augmentation strategies were explored to improve the robustness of the model against environmental variations; and a fine-tuning policy was proposed to further improve the subject-specific accuracy. The decoding model was finally tested in three scenarios: a robotic arm/hand system performing daily-living tasks (pouring, screwing, etc.), a supernumerary robotic hand playing blocks-building, and a virtual prosthesis conducting motor rehabilitation training.

This article aims to solve the problem of formation control of mobile robots based on image and provide a low cost as well as ease of implementation solution for mobile robots relying merely on a monocular camera under field-of-view (FOV) constraints. A low complexity image-based visual servo (IBVS) controller is proposed, which can achieve the desired relative position on the image plane and solve the FOV constraints without the feature depth and leader's velocities information. To facilitate the control design, a state transformation is first performed to decouple the visual motion kinematics. Then, an error transformation is introduced to handle the FOV constraints and performance specifications are incorporated in the error transformation to achieve the predefined control performance. Finally, a simple static controller is derived using only information from images, and the stability of the uncertain system with unknown control direction/coefficients under the given performance control condition is analyzed. The effectiveness and performance of the proposed visual servoing controller can be illustrated using both simulations and experiments.

Multi-agent systems (MASs) can autonomously learn to solve previously unknown tasks by means of each agent's individual intelligence as well as by collaborating and exploiting collective intelligence. This article considers a group of autonomous agents learning to track the same given reference trajectory in a possibly small number of trials. We propose a novel collective learning control method that combines iterative learning control (ILC) with a collective update strategy. We derive conditions for desirable convergence properties of such systems. We show that the proposed method allows the collective to combine the advantages of the agents' individual learning strategies and thereby overcomes trade-offs and limitations of single-agent ILC. This benefit is achieved by designing a heterogeneous collective, i.e., a different learning law is assigned to each agent. All theoretical results are confirmed in simulations and experiments with two-wheeled-inverted-pendulum robots (TWIPRs) that jointly learn to perform the desired maneuver.

Accurate vehicle sideslip angle estimation is crucial for vehicle stability control. In this paper, an enabling event-triggered sideslip angle estimator is proposed by using the kinematic information from a low-cost Global Positioning System (GPS) and an on-board Inertial Measurement Unit (IMU). First, a preliminary vehicle sideslip angle is derived using the heading angle of GPS and the yaw rate of IMU, and an event-triggered mechanism is proposed to eliminate the accumulative estimation error. The algorithm convergence is guaranteed through theoretical deduction. Second, a longitudinal and a lateral vehicle velocity are obtained using the preliminary vehicle sideslip angle and the measured GPS velocity and their kinematic relationship, based on which a multi-sensor fusion and a multi-step Kalman filter scheme are respectively presented to realize longitudinal and lateral vehicle velocity estimation. By doing this, the update frequency and estimation accuracy of the vehicle sideslip angle estimate can be further improved to meet the requirement of online implementation. Finally, the effectiveness and reliability of the proposed scheme are verified under comprehensive driving conditions through both hardware-in-loop (HIL) and field tests. The results show that the proposed event-triggered sideslip angle estimator has a mean estimation error of 0.029 deg and of 0.14 deg in the HIL and field tests, exhibiting better estimation accuracy, reliability and real-time performance compared with other typical estimators.

Localization is an essential technique in mobile robotics. In a complex environment, it is necessary to fuse different localization modules to obtain more robust results, in which the error model plays a paramount role. However, exteroceptive sensor-based odometries(ESOs), such as LiDAR/visual odometry, often deliver results with scene-related error, which is difficult to model accurately. To address this problem, this research designs a scene-aware error model for ESO, based on which a multimodal localization fusion framework is developed. In addition, an end-to-end learning method is proposed to train this error model using sparse global poses such as results from global positioning system(GPS) and inertial measurement unit(IMU). The proposed method is realized for error modeling of LiDAR/visual odometry, and the results are fused with dead reckoning to examine the performance of vehicle localization. Experiments are conducted using both simulation and real-world data of experienced and unexperienced environments, and the experimental results demonstrate that with the learned scene-aware error models, vehicle localization accuracy can be largely improved and shows adaptiveness in unexperienced scenes.

Potholes are one of the most common forms of road damage, which can severely affect driving comfort, road safety, and vehicle condition. Pothole detection is typically performed by either structural engineers or certified inspectors. However, this task is not only hazardous for the personnel but also extremely time consuming. This article presents an efficient pothole detection algorithm based on road disparity map estimation and segmentation. We first incorporate the stereo rig roll angle into shifting distance calculation to generalize perspective transformation. The road disparities are then efficiently estimated using semiglobal matching. A disparity map transformation algorithm is then performed to better distinguish the damaged road areas. Subsequently, we utilize simple linear iterative clustering to group the transformed disparities into a collection of superpixels. The potholes are finally detected by finding the superpixels, whose intensities are lower than an adaptively determined threshold. The proposed algorithm is implemented on an NVIDIA RTX 2080 Ti GPU in CUDA. The experimental results demonstrate that our proposed road pothole detection algorithm achieves state-of-the-art accuracy and efficiency.

Pedestrians represent acutely complex and agile targets whose accident prevention has the highest priority concerning highly automated vehicles in urban traffic. Consequently, detection and tracking methods are necessary to determine motion behavior quickly and with high precision. With advancing microelectronics, powerful radar sensors and novel antenna structures are available for pedestrian detection systems, which can utilize unique features such as micro-Doppler signatures in several dimensions and high resolution. This paper presents a micro-Doppler based leg tracking framework that enables behavioral indications in a few sensor cycles. An adaptive motion model representing the kinematic locomotion of a human is derived to propagate the foot motion. Joint probabilistic data association assigns the detections to the respective leg and allows its temporal filtering of the position, micro-Doppler velocity, and kinematic parameters using an extended Kalman filter yielding real-time implementation capability. The presented approach provides the entire signal chain from raw radar data processing to extended multi-object state estimation based on highly relevant safety-critical motion maneuvers measured with radar parameterization corresponding to current serial short-range sensors.

In this paper, we explore the challenging 1-to-N map matching problem, which exploits a compact description of map data, to improve the scalability of map matching techniques used by various robot vision tasks. We propose a first method explicitly aimed at fast succinct map matching, which consists only of map-matching subtasks. These tasks include offline map matching attempts to find a compact part-based scene model that effectively explains each map using fewer larger parts. The tasks also include an online map matching attempt to efficiently find correspondence between the part-based maps. Our part-based scene modeling approach is unsupervised and uses common pattern discovery (CPD) between the input and known reference maps. This enables a robot to learn a compact map model without human intervention. We also present a practical implementation that uses the state-of-the-art CPD technique of randomized visual phrases (RVP) with a compact bounding box (BB) based part descriptor, which consists of keypoint and descriptor BBs. The results of our challenging map-matching experiments, which use a publicly available radish dataset, show that the proposed approach achieves successful map matching with significant speedup and a compact description of map data that is tens of times more compact. Although this paper focuses on the standard 2D point-set map and the BB-based part representation, we believe our approach is sufficiently general to be applicable to a broad range of map formats, such as the 3D point cloud map, as well as to general bounding volumes and other compact part representations., Offprint of IROS2013 paper

In this study, we present a new experiment in order to study the Lombard effect in telepresence robotics. In this experiment, one person talks with a robot controled remotely by someone in a different room. The remote pilot (R) is immersed in both environments, while the local interlocutor (L) interacts directly with the robot. In this context, the position of the noise source, in the remote or in the local room, may modify the subjects’ voice adaptations. In order to study in details this phenomenon, we propose four particular conditions: no added noise, noise in room R heard only by R, virtual noise in room L heard only by R, and noise in room L heard by both R and L. We measured the variations of maximum intensity in order to quantify the Lombard effect. Our results show that there is indeed a modification of voice intensity in all noisy conditions. However, the amplitude of this modification varies depending on the condition.

Online recognition of gestures is critical for intuitive human-robot interaction (HRI) and further push collaborative robotics into the market, making robots accessible to more people. The problem is that it is difficult to achieve accurate gesture recognition in real unstructured environments, often using distorted and incomplete multisensory data. This paper introduces an HRI framework to classify large vocabularies of interwoven static gestures (SGs) and dynamic gestures (DGs) captured with wearable sensors. DG features are obtained by applying data dimensionality reduction to raw data from sensors (resampling with cubic interpolation and principal component analysis). Experimental tests were conducted using the UC2017 hand gesture dataset with samples from eight different subjects. The classification models show an accuracy of 95.6% for a library of 24 SGs with a random forest and 99.3% for 10 DGs using artificial neural networks. These results compare equally or favorably with different commonly used classifiers. Long short-term memory deep networks achieved similar performance in online frame-by-frame classification using raw incomplete data, performing better in terms of accuracy than static models with specially crafted features, but worse in training and inference time. The recognized gestures are used to teleoperate a robot in a collaborative process that consists in preparing a breakfast meal.

This paper presents the technique of flex-and-flip manipulation. It is suitable for grasping thin, flexible linear objects lying on a flat surface. During the manipulation process, the object is first flexed by a robotic gripper whose fingers are placed on top of it, and later the increased internal energy of the object helps the gripper obtain a stable pinch grasp while the object flips into the space between the fingers. The dynamic interaction between the flexible object and the gripper is elaborated by analyzing how energy is exchanged. We also discuss the condition on friction to prevent loss of contact. Our flex-and-flip manipulation technique can be implemented with open-loop control and lends itself to underactuated, compliant finger mechanism. A set of experiments in robotic page turning performed with our customized hardware and software system demonstrates the effectiveness and robustness of the manipulation technique., 6 pages,8 figures, 2 tables