Search

Quality assessment and aesthetics assessment aim to evaluate the perceived quality and aesthetics of visual content. Current learning-based methods suffer greatly from the scarcity of labeled data and usually perform sub-optimally in terms of generalization. Although masked image modeling (MIM) has achieved noteworthy advancements across various high-level tasks (e.g., classification, detection etc.). In this work, we take on a novel perspective to investigate its capabilities in terms of quality- and aesthetics-awareness. To this end, we propose Quality- and aesthetics-aware pretraining (QPT V2), the first pretraining framework based on MIM that offers a unified solution to quality and aesthetics assessment. To perceive the high-level semantics and fine-grained details, pretraining data is curated. To comprehensively encompass quality- and aesthetics-related factors, degradation is introduced. To capture multi-scale quality and aesthetic information, model structure is modified. Extensive experimental results on 11 downstream benchmarks clearly show the superior performance of QPT V2 in comparison with current state-of-the-art approaches and other pretraining paradigms. Code and models will be released at \url{https://github.com/KeiChiTse/QPT-V2}., Comment: 8 pages, 6 figures

Large Language Models (LLMs) have been widely utilized to perform complex robotic tasks. However, handling external disturbances during tasks is still an open challenge. This paper proposes a novel method to achieve robotic adaptive tasks based on LLMs and Behavior Trees (BTs). It utilizes ChatGPT to reason the descriptive steps of tasks. In order to enable ChatGPT to understand the environment, semantic maps are constructed by an object recognition algorithm. Then, we design a Parser module based on Bidirectional Encoder Representations from Transformers (BERT) to parse these steps into initial BTs. Subsequently, a BTs Update algorithm is proposed to expand the initial BTs dynamically to control robots to perform adaptive tasks. Different from other LLM-based methods for complex robotic tasks, our method outputs variable BTs that can add and execute new actions according to environmental changes, which is robust to external disturbances. Our method is validated with simulation in different practical scenarios., Comment: 7 pages, 11figures, WILL PUBLISHED ON ICRA 2024

Deformable linear object (DLO) manipulation is needed in many fields. Previous research on deformable linear object (DLO) manipulation has primarily involved parallel jaw gripper manipulation with fixed grasping positions. However, the potential for dexterous manipulation of DLOs using an anthropomorphic hand is under-explored. We present DexDLO, a model-free framework that learns dexterous dynamic manipulation policies for deformable linear objects with a fixed-base dexterous hand in an end-to-end way. By abstracting several common DLO manipulation tasks into goal-conditioned tasks, our DexDLO can perform these tasks, such as DLO grabbing, DLO pulling, DLO end-tip position controlling, etc. Using the Mujoco physics simulator, we demonstrate that our framework can efficiently and effectively learn five different DLO manipulation tasks with the same framework parameters. We further provide a thorough analysis of learned policies, reward functions, and reduced observations for a comprehensive understanding of the framework., Comment: 7 pages, 8 figures, submitted to ICRA 2024

Bagging is an essential skill that humans perform in their daily activities. However, deformable objects, such as bags, are complex for robots to manipulate. This paper presents an efficient learning-based framework that enables robots to learn bagging. The novelty of this framework is its ability to perform bagging without relying on simulations. The learning process is accomplished through a reinforcement learning algorithm introduced in this work, designed to find the best grasping points of the bag based on a set of compact state representations. The framework utilizes a set of primitive actions and represents the task in five states. In our experiments, the framework reaches a 60 % and 80 % of success rate after around three hours of training in the real world when starting the bagging task from folded and unfolded, respectively. Finally, we test the trained model with two more bags of different sizes to evaluate its generalizability., Comment: IET Cyber-Systems and Robotics

Learning from Demonstration (LfD) enables robots to acquire versatile skills by learning motion policies from human demonstrations. It endows users with an intuitive interface to transfer new skills to robots without the need for time-consuming robot programming and inefficient solution exploration. During task executions, the robot motion is usually influenced by constraints imposed by environments. In light of this, task-parameterized LfD (TP-LfD) encodes relevant contextual information into reference frames, enabling better skill generalization to new situations. However, most TP-LfD algorithms typically require multiple demonstrations across various environmental conditions to ensure sufficient statistics for a meaningful model. It is not a trivial task for robot users to create different situations and perform demonstrations under all of them. Therefore, this paper presents a novel algorithm to learn skills from few demonstrations. By leveraging the reference frame weights that capture the frame importance or relevance during task executions, our method demonstrates excellent skill acquisition performance, which is validated in real robotic environments., Comment: Accepted by ISER. For the experiment video, see https://youtu.be/JpGjk4eKC3o

Deformable Object Manipulation (DOM) is an important field of research as it contributes to practical tasks such as automatic cloth handling, cable routing, surgical operation, etc. Perception is considered one of the major challenges in DOM due to the complex dynamics and high degree of freedom of deformable objects. In this paper, we develop a novel image-processing algorithm based on Gabor filters to extract useful features from cloth, and based on this, devise a strategy for cloth flattening tasks. We also evaluate the overall framework experimentally and compare it with three human operators. The results show that our algorithm can determine the direction of wrinkles on the cloth accurately in simulation as well as in real robot experiments. Furthermore, our dewrinkling strategy compares favorably to baseline methods. The experiment video is available on https://sites.google.com/view/robotic-fabric-flattening/home, Comment: Accepted by the 18th International Symposium on Experimental Robotics (ISER 2023)

Tracking tasks based on deep neural networks have greatly improved with the emergence of Siamese trackers. However, the appearance of targets often changes during tracking, which can reduce the robustness of the tracker when facing challenges such as aspect ratio change, occlusion, and scale variation. In addition, cluttered backgrounds can lead to multiple high response points in the response map, leading to incorrect target positioning. In this paper, we introduce two transformer-based modules to improve Siamese tracking called DASTSiam: the spatio-temporal (ST) fusion module and the Discriminative Augmentation (DA) module. The ST module uses cross-attention based accumulation of historical cues to improve robustness against object appearance changes, while the DA module associates semantic information between the template and search region to improve target discrimination. Moreover, Modifying the label assignment of anchors also improves the reliability of the object location. Our modules can be used with all Siamese trackers and show improved performance on several public datasets through comparative and ablation experiments.

Developing physically assistive robots capable of dressing assistance has the potential to significantly improve the lives of the elderly and disabled population. However, most robotics dressing strategies considered a single robot only, which greatly limited the performance of the dressing assistance. In fact, healthcare professionals perform the task bimanually. Inspired by them, we propose a bimanual cooperative scheme for robotic dressing assistance. In the scheme, an interactive robot joins hands with the human thus supporting/guiding the human in the dressing process, while the dressing robot performs the dressing task. We identify a key feature that affects the dressing action and propose an optimal strategy for the interactive robot using the feature. A dressing coordinate based on the posture of the arm is defined to better encode the dressing policy. We validate the interactive dressing scheme with extensive experiments and also an ablation study. The experiment video is available on https://sites.google.com/view/bimanualassitdressing/home

Previous work has shown that 3D point cloud classifiers can be vulnerable to adversarial examples. However, most of the existing methods are aimed at white-box attacks, where the parameters and other information of the classifiers are known in the attack, which is unrealistic for real-world applications. In order to improve the attack performance of the black-box classifiers, the research community generally uses the transfer-based black-box attack. However, the transferability of current 3D attacks is still relatively low. To this end, this paper proposes Scale and Shear (SS) Attack to generate 3D adversarial examples with strong transferability. Specifically, we randomly scale or shear the input point cloud, so that the attack will not overfit the white-box model, thereby improving the transferability of the attack. Extensive experiments show that the SS attack proposed in this paper can be seamlessly combined with the existing state-of-the-art (SOTA) 3D point cloud attack methods to form more powerful attack methods, and the SS attack improves the transferability over 3.6 times compare to the baseline. Moreover, while substantially outperforming the baseline methods, the SS attack achieves SOTA transferability under various defenses. Our code will be available online at https://github.com/cuge1995/SS-attack, Comment: 10 pages

Most object manipulation strategies for robots are based on the assumption that the object is rigid (i.e., with fixed geometry) and the goal's details have been fully specified (e.g., the exact target pose). However, there are many tasks that involve spatial relations in human environments where these conditions may be hard to satisfy, e.g., bending and placing a cable inside an unknown container. To develop advanced robotic manipulation capabilities in unstructured environments that avoid these assumptions, we propose a novel long-horizon framework that exploits contrastive planning in finding promising collaborative actions. Using simulation data collected by random actions, we learn an embedding model in a contrastive manner that encodes the spatio-temporal information from successful experiences, which facilitates the subgoal planning through clustering in the latent space. Based on the keypoint correspondence-based action parameterization, we design a leader-follower control scheme for the collaboration between dual arms. All models of our policy are automatically trained in simulation and can be directly transferred to real-world environments. To validate the proposed framework, we conduct a detailed experimental study on a complex scenario subject to environmental and reachability constraints in both simulation and real environments., Comment: 11 pages, 13 figures

This paper presents a topology optimization approach for the surface flows on variable design domains. Via this approach, the matching between the pattern of a surface flow and the 2-manifold used to define the pattern can be optimized, where the 2-manifold is implicitly defined on another fixed 2-manifold named as the base manifold. The fiber bundle topology optimization approach is developed based on the description of the topological structure of the surface flow by using the differential geometry concept of the fiber bundle. The material distribution method is used to achieve the evolution of the pattern of the surface flow. The evolution of the implicit 2-manifold is realized via a homeomorphous map. The design variable of the pattern of the surface flow and that of the implicit 2-manifold are regularized by two sequentially implemented surface-PDE filters. The two surface-PDE filters are coupled, because they are defined on the implicit 2-manifold and base manifold, respectively. The surface Navier-Stokes equations, defined on the implicit 2-manifold, are used to describe the surface flow. The fiber bundle topology optimization problem is analyzed using the continuous adjoint method implemented on the first-order Sobolev space. Several numerical examples have been provided to demonstrate this approach, where the combination of the viscous dissipation and pressure drop is used as the design objective.

In extending fast digital clock synchronization to the bounded-delay model, the expected constant time Byzantine pulse resynchronization problem is investigated. In this problem, the synchronized state of the system should not only be deterministically maintained but be reached from arbitrary states with expected constant time in the presence of Byzantine faults. An intuitive geometric representation of the problem is introduced, with which the classical approximate agreement, randomized Byzantine agreement, and random walk are integrated with some geometric operations. Efficient realizations are also provided for practical uses. Compared with the state-of-the-art solutions, the assumed common pulses need not be regularly generated, the message complexity can be lowered as approximate agreement, and the expected stabilization time is optimal. With this, the provided solution can efficiently convert irregularly generated common pulses to self-stabilizing Byzantine pulse synchronization., Comment: 9 pages, 4 figures, submitted to Elsevier

For reaching dependable high-precision clock synchronization (CS) upon IoT networks, the distributed CS paradigm adopted in ultra-high reliable systems and the master-slave CS paradigm adopted in high-performance but unreliable systems are integrated. Meanwhile, traditional internal clock synchronization is also integrated with external time references to achieve efficient stabilization. Low network connectivity, low complexity, high precision, and high reliability are all considered. To tolerate permanent failures, the Byzantine CS is integrated with the common CS protocols. To tolerate transient failures, the self-stabilizing Byzantine CS is also extended upon open-world IoT networks. With these, the proposed intro-stabilizing Byzantine CS solution can establish and maintain synchronization with arbitrary initial states in the presence of permanent Byzantine faults. With the formal analysis and numerical simulations, it is shown that the best of the CS solutions provided for the ultra-high reliable systems and the high-performance unreliable systems can be well integrated upon IoT networks to derive dependable high-precision CS even across the traditional closed safety-boundary., Comment: 26 pages, 18 figures

For reaching fast and efficient self-stabilizing Byzantine pulse synchronization (SSBPS) upon the bounded-delay message-passing networks, we consider the peaceable SSBPS problem where the resource occupation in the stabilized system is required to be as sparse as possible and the stabilization of the system is required to be as fast as possible. To solve it, the decoupled absorption process and emergency process are investigated under a general framework. The constant-time two-stage absorption process and more than one kind of emergency process are provided in integrating the merits of temporal trails, approximate agreements, deterministic and randomized Byzantine agreements, and self-stabilizing protocols. With this, not only SSBPS but self-stabilizing Byzantine clock synchronization can be fast established and efficiently maintained. In the optimal-resilient basic solutions, the deterministic linear stabilization time and the stabilized resource occupation are all optimized to those of the underlying primitives, which is optimal in the presence of Byzantine faults under the classical static adversary. In the hybrid solutions, faster stabilization can be expected with a worst-case deterministic stabilization time. In all solutions, the stabilized resource occupation is at most at the order of approximate agreement, which is a good property in considering real-world ultra-high-reliable hard-real-time applications where pulse synchronization is expected to be established before all upper-layer functions and to be efficiently maintained when the upper-layer functions are in service., Comment: 23 pages, 3 figures

Robust pulse synchronization is fundamental in constructing reliable synchronous applications in wired and wireless distributed systems. In wired systems, self-stabilizing Byzantine pulse synchronization aims for synchronizing fault-prone distributed components with arbitrary initial states in bounded-delay message-passing systems. In wireless systems, fault-tolerant synchronization of pulse-coupled oscillators is also built for a similar goal but often works under specific system restrictions, such as low computation power, low message complexity, and anonymous physical pulses whose senders cannot be identified by the receivers. These restrictions often prevent us from constructing high-reliable wireless synchronous applications. This paper tries to break barriers between bounded-delay message-passing systems and classical pulse-coupled oscillators by introducing a new fault-tolerant synchronization scheme for the so-called anonymous bounded-delay pulsing systems in the presence of indeterministic communication delays and inconsistent faults. For low computation power and low message complexity, instead of involving in consensus-based primitives, the proposed synchronization scheme integrates the so-called discrete mean-fields, planar random walks, and some additional easy operations in utilizing only sparsely generated anonymous pulses. For fault-tolerance, we show that a square-root number of faulty oscillators can be tolerated by utilizing planar random walks in anonymous pulse synchronization. For self-stabilization, we show that the stabilization can be reached in an expected constant number of observing windows in anonymous bounded-delay pulsing systems with the pulsing-frequency restriction., Comment: 19 pages, 5 figures

For reaching efficient deterministic synchronous Byzantine agreement upon partially connected networks, the traditional broadcast primitive is extended and integrated with a general framework. With this, the Byzantine agreement is extended to fully connected bipartite networks and some bipartite bounded-degree networks. The complexity of the Byzantine agreement is lowered and optimized with the so-called Byzantine-levers under a general system structure. Some bipartite simulation of the butterfly networks and some finer properties of bipartite bounded-degree networks are also provided for building efficient incomplete Byzantine agreement under the same system structure. It shows that efficient real-world Byzantine agreement systems can be built with the provided algorithms with sufficiently high system assumption coverage. Meanwhile, the proposed bipartite solutions can improve the dependability of the systems in some open, heterogeneous, and even antagonistic environments., Comment: 24 pages, 5 figures

For tolerating Byzantine faults of both the terminal and communication components in self-stabilizing clock synchronization, the two-dimensional self-stabilizing Byzantine-fault-tolerant clock synchronization problem is investigated and solved. By utilizing the time-triggered (TT) stage provided in the underlying networks as TT communication windows, the approximate agreement, hopping procedure, and randomized grandmasters are integrated into the overall solution. It is shown that with partitioning the communication components into $3$ arbitrarily connected subnetworks, efficient synchronization can be achieved with one such subnetwork and less than $1/3$ terminal components being Byzantine. Meanwhile, the desired stabilization can be reached for the specific networks in one or several seconds with high probabilities. This helps in developing various distributed hard-real-time systems with stringent time, resources, and safety requirements., Comment: 8 pages, 3 figures

To improve the overall efficiency and reliability of Byzantine protocols in large sparse networks, we propose a new system assumption for developing multi-scale fault-tolerant systems, with which several kinds of multi-scale Byzantine protocols are developed in large sparse networks with high system assumption coverage. By extending the traditional Byzantine adversary to the multi-scale adversaries, it is shown that efficient deterministic Byzantine broadcast and Byzantine agreement can be built in logarithmic-degree networks. Meanwhile, it is shown that the multi-scale adversary can make a finer trade-off between the system assumption coverage and the overall efficiency of the Byzantine protocols, especially when a small portion of the low-layer small-scale protocols are allowed to fail arbitrarily. With this, efficient Byzantine protocols can be built in large sparse networks with high system reliability., Comment: 8 pages, 1 figure

The authenticated broadcast is simulated in the bounded-degree networks to provide efficient broadcast primitives for building efficient higher-layer Byzantine protocols. A general abstraction of the relay-based broadcast system is introduced, in which the properties of the relay-based broadcast primitives are generalized. With this, fault-tolerant propagation is proposed as a building block of the broadcast primitives. Meanwhile, complementary systems are proposed in complementing fault-tolerant propagation and localized communication. Analysis shows that efficient fault-tolerant propagation can be built with sufficient initiation areas. Meanwhile, by integrating fault-tolerant propagation and localized communication, efficient broadcast primitives can be built in bounded-degree networks., Comment: 8 pages, 2 figures

Message propagation is fundamental in constructing distributed systems upon sparsely connected communication networks. For providing easy message propagation primitives, the mutual-exclusive propagation (MEP) of one-bit messages is investigated with a simplified discrete system model. Inspired by natural propagation systems, efficient self-stabilizing periodic MEP processes are proposed with the MEP systems. For handling the worst-case scenarios, the MEP systems are generally analyzed with formal proofs upon arbitrarily connected networks. It shows that the MEP systems can be stabilized with arbitrary initial system states within a short bounded time. Meanwhile, the numeric simulation shows that the propagation errors can be significantly reduced if the message delays are randomly distributed. Propagation patterns are also discussed in further deriving finer propagation results in the MEP systems. It shows that bounded clock drifts and some benign faults can be well handled in the MEP systems. Featured by its very simple mechanism, the proposed MEP primitive can be employed as a practical building block of upper-layer self-stabilizing synchronous systems., Comment: 12 pages, 7 figures

Estimating 6D object pose from an RGB image is important for many real-world applications such as autonomous driving and robotic grasping. Recent deep learning models have achieved significant progress on this task but their robustness received little research attention. In this work, for the first time, we study adversarial samples that can fool deep learning models with imperceptible perturbations to input image. In particular, we propose a Unified 6D pose estimation Attack, namely U6DA, which can successfully attack several state-of-the-art (SOTA) deep learning models for 6D pose estimation. The key idea of our U6DA is to fool the models to predict wrong results for object instance localization and shape that are essential for correct 6D pose estimation. Specifically, we explore a transfer-based black-box attack to 6D pose estimation. We design the U6DA loss to guide the generation of adversarial examples, the loss aims to shift the segmentation attention map away from its original position. We show that the generated adversarial samples are not only effective for direct 6D pose estimation models, but also are able to attack two-stage models regardless of their robust RANSAC modules. Extensive experiments were conducted to demonstrate the effectiveness, transferability, and anti-defense capability of our U6DA on large-scale public benchmarks. We also introduce a new U6DA-Linemod dataset for robustness study of the 6D pose estimation task. Our codes and dataset will be available at \url{https://github.com/cuge1995/U6DA}., Comment: 15 pages

Deep neural networks (DNNs) have been shown to be vulnerable to adversarial attacks. Recently, 3D adversarial attacks, especially adversarial attacks on point clouds, have elicited mounting interest. However, adversarial point clouds obtained by previous methods show weak transferability and are easy to defend. To address these problems, in this paper we propose a novel point cloud attack (dubbed AOF) that pays more attention on the low-frequency component of point clouds. We combine the losses from point cloud and its low-frequency component to craft adversarial samples. Extensive experiments validate that AOF can improve the transferability significantly compared to state-of-the-art (SOTA) attacks, and is more robust to SOTA 3D defense methods. Otherwise, compared to clean point clouds, adversarial point clouds obtained by AOF contain more deformation than outlier., Comment: 8 pages

Moving away from repetitive tasks, robots nowadays demand versatile skills that adapt to different situations. Task-parameterized learning improves the generalization of motion policies by encoding relevant contextual information in the task parameters, hence enabling flexible task executions. However, training such a policy often requires collecting multiple demonstrations in different situations. To comprehensively create different situations is non-trivial thus renders the method less applicable to real-world problems. Therefore, training with fewer demonstrations/situations is desirable. This paper presents a novel concept to augment the original training dataset with synthetic data for policy improvements, thus allows learning task-parameterized skills with few demonstrations., Comment: Accepted by the IEEE Robotics and Automation Letters

The robotic manipulation of composite rigid-deformable objects (i.e. those with mixed non-homogeneous stiffness properties) is a challenging problem with clear practical applications that, despite the recent progress in the field, it has not been sufficiently studied in the literature. To deal with this issue, in this paper we propose a new visual servoing method that has the capability to manipulate this broad class of objects (which varies from soft to rigid) with the same adaptive strategy. To quantify the object's infinite-dimensional configuration, our new approach computes a compact feedback vector of 2D contour moments features. A sliding mode control scheme is then designed to simultaneously ensure the finite-time convergence of both the feedback shape error and the model estimation error. The stability of the proposed framework (including the boundedness of all the signals) is rigorously proved with Lyapunov theory. Detailed simulations and experiments are presented to validate the effectiveness of the proposed approach. To the best of the author's knowledge, this is the first time that contour moments along with finite-time control have been used to solve this difficult manipulation problem.

Deformable object manipulation (DOM) is an emerging research problem in robotics. The ability to manipulate deformable objects endows robots with higher autonomy and promises new applications in the industrial, services, and healthcare sectors. However, compared to rigid object manipulation, the manipulation of deformable objects is considerably more complex, and is still an open research problem. Addressing DOM challenges demand breakthroughs in almost all aspects of robotics, namely hardware design, sensing, (deformation) modeling, planning, and control. In this article, we review recent advances and highlight the main challenges when considering deformation in each sub-field. A particular focus of our paper lies in the discussions of these challenges and proposing future directions of research.