Search

The increasing vessel size and automation level have shifted the productivity bottleneck of automated container terminals from the terminal side to the yard side. Operating an automated container terminal (ACT) yard with a big number of automated guided vehicles (AGV) is challenging due to the complexity and dynamics of the system, severely affecting the operational efficiency and energy use efficiency. In this paper, a hybrid multi-AGV scheduling algorithm is proposed to minimise the energy consumption and the total makespan of AGVs in an ACT yard. This framework first models the AGV scheduling process as a Markov decision process (MDP). Furthermore, a novel scheduling algorithm called MDAS is proposed based on multi-agent deep deterministic policy gradient (MADDPG) to facilitate online real-time scheduling decision-making. Finally, simulation experiments show that the proposed method can effectively enhance the operational efficiency and energy use performance of AGVs in ACT yards of various scales by comparing with benchmarking methods. [ABSTRACT FROM AUTHOR]

The cloud-edge collaborative manufacturing system (CCMS) connects distributed factories to a cloud centre through cloud-edge collaborative communication, introducing both opportunities and challenges to conventional dynamic job scheduling. Enhancing each factory's scheduling performance by sharing general scheduling knowledge among heterogeneous factories under the consideration of data privacy protection remains challenging. To this end, this paper proposes to solve the dynamic job scheduling in the context of CCMS with a novel federated deep reinforcement learning (FDRL) approach. Within each factory, the scheduling objective involves minimising the makespan and energy consumption, accounting for machine warm-up procedures and real-time dynamics. To handle heterogeneous policy structures, we aggregate their hidden parameters through FDRL, with states, actions, and rewards designed to facilitate the aggregation. The two-phase algorithm, comprising iterative local training and global aggregation, trains the scheduling policies. Constraint items are introduced to the loss functions to smooth local training, and the global aggregation considers production scales and obtained objectives. The proposed approach enhances the solution quality and generalisation of each factory's scheduling policy without exposing original production data. Numerical experiments conducted on sixty scheduling instances validate the superiority of the proposed approach compared to twelve dynamic scheduling methods. Compared to independently trained DRL-based approaches, the proposed FDRL-based approach achieves up to an 8.9% reduction in makespan and a 22.3% decrease in energy consumption through knowledge sharing. [ABSTRACT FROM AUTHOR]

One of the main challenges in operating multistate manufacturing systems (MMSs) is maintaining stable and robust production against various disruptions. Therefore, an urgent need exists for an operation and maintenance (O&M) method that optimises MMS resilience, i.e. the capability of withstanding or recover from disruptions of various sources. Consequently, this study proposes an integrated resilience-oriented production and maintenance scheduling approach for MMSs. This approach enhances MMS resilience by reinforcing its adaptivity to the variation in production requirements. Based on a conceptual investigation of operational uncertainty and its mechanism of disruption, this study devotes to (i) formulating a performance loss-based resilience measurement with consideration of operational uncertainties and (ii) proposing a reinforcement learning-based approach to schedule MMS operation and maintenance activities for MMS components of different performance states. An industrial case study of a ferrite phase shifter manufacturing system is subsequently conducted to validate the proposed approach. Results demonstrate the effectiveness of the proposed approach in the resilience optimisation of MMSs. [ABSTRACT FROM AUTHOR]

Nowadays industrial robots have been increasingly widely deployed across various sectors and assembly tasks are seen as one of the dominant application fields. The assembly tasks, as the critical process in manufacturing, are still challenging for the robot because of the complex contact state between the robot and the environment (i.e. assembly components). In the assembly task, the robot should switch its controller from a non-contact mode to a contact-rich mode when the contact condition changes, where compliant behaviour is necessary for robustness to uncertain contact and safety of physical interaction. This paper proposes a deep reinforcement learning (DRL) method to achieve such compliance using variable stiffness compliant control. Concretely, a Cartesian compliant controller is built on a virtual dynamics model with a variable non-diagonal stiffness matrix to derive desired motion that reacts with the external force/torque. Upon that, a deep deterministic policy gradient (DDPG)-based agent is deployed to fine-tune such a non-diagonal stiffness matrix. After error-and-trial learning, robots can handle changes in contact state in assembly tasks without any pause and switching its controller mode, thus increasing task efficiency and compliance. The simulation and experiments show that our method allows robots to complete assembly tasks safely and efficiently under noisy observation. [ABSTRACT FROM AUTHOR]

This document discusses the importance of carbon reduction engineering in combating global climate change. It highlights the role of production and supply chain management in achieving carbon neutrality and reducing carbon emissions. The document presents 20 selected research papers that cover various aspects of production optimization in carbon reduction engineering, including carbon reduction in the production process, low-carbon and sustainable supply chains, assessment and optimization methodologies, and policy instruments. The papers provide insights into topics such as scheduling optimization, renewable energy production, carbon emission efficiency, sustainable consumption, government subsidizing arrangements, and the impact of policy instruments on carbon reduction. The document expresses gratitude to the authors and reviewers for their contributions and acknowledges the support of the International Journal of Production Research. [Extracted from the article]

The semiconductor industry is a resource-intensive sector that heavily relies on energy, water, chemicals, and raw materials. Within the semiconductor manufacturing process, the diffusion furnace, ion implantation machine, and plasma etching machine exhibit high energy demands or operate at extremely high temperatures, resulting in significant electricity consumption, which is usually carbon-intensive. To address energy conservation concerns, the industry adopts batch production technology, which allows for the simultaneous processing of multiple products. The energy-efficient parallel batch scheduling problem arises from the need to optimise product grouping and sequencing. In contrast to existing heuristics, meta-heuristics, and exact algorithms, this paper introduces the Deep Q-Network (DQN) algorithm as a novel approach to address the proposed problem. The DQN algorithm is built upon the agent's systematic learning of scheduling rules, thereby enabling it to offer guidance for online decision-making regarding the grouping and sequencing of products. The efficacy of the algorithm is substantiated through extensive computational experiments. [ABSTRACT FROM AUTHOR]

This study conducts a comprehensive analysis of deep reinforcement learning (DRL) algorithms applied to supply chain inventory management (SCIM), which consists of a sequential decision-making problem focussed on determining the optimal quantity of products to produce and ship across multiple capacitated local warehouses over a specific time horizon. In detail, we formulate the problem as a Markov decision process for a divergent two-echelon inventory control system characterised by stochastic and seasonal demand, also presenting a balanced allocation rule designed to prevent backorders in the first echelon. Through numerical experiments, we evaluate the performance of state-of-the-art DRL algorithms and static inventory policies in terms of both cost minimisation and training time while varying the number of local warehouses and product types and the length of replenishment lead times. Our results reveal that the Proximal Policy Optimization algorithm consistently outperforms other algorithms across all experiments, proving to be a robust method for tackling the SCIM problem. Furthermore, the (s, Q)-policy stands as a solid alternative, offering a compromise in performance and computational efficiency. Lastly, this study presents an open-source software library that provides a customisable simulation environment for addressing the SCIM problem, utilising a wide range of DRL algorithms and static inventory policies. [ABSTRACT FROM AUTHOR]

Shop scheduling is deeply involved in manufacturing. In order to improve the efficiency of scheduling and fit dynamic scenarios, many Deep Reinforcement Learning (DRL) methods are studied to solve scheduling problems like job shop and flow shop. But most studies focus on using the latest algorithms while ignoring that the environment plays an important role in agent learning. In this paper, we design an effective, robust and size-agnostic environment for job shop scheduling. The proposed design of environment uses centralised training and decentralised execution (CTDE) to implement a multi-agent architecture. Together with the observation space we design, environmental information that is irrelevant to the current decision is eliminated as much as possible. The proposed action space enlarges the decision space of agents, which performs better than the traditional way. Finally, Soft Actor-Critic (SAC) algorithm is adapted to learning within this environment. By comparing with traditional scheduling rules, other reinforcement learning algorithms, and relevant literature, the superiority of the results obtained in this study is demonstrated. [ABSTRACT FROM AUTHOR]

Policy-guided Monte Carlo is an adaptive method to simulate classical interacting systems. It adjusts the proposal distribution of the Metropolis–Hastings algorithm to maximize the sampling efficiency, using a formalism inspired by reinforcement learning. In this work, we first extend the policy-guided method to deal with a general state space, comprising, for instance, both discrete and continuous degrees of freedom, and then apply it to a few paradigmatic models of glass-forming mixtures. We assess the efficiency of a set of physically inspired moves whose proposal distributions are optimized through on-policy learning. Compared to conventional Monte Carlo methods, the optimized proposals are two orders of magnitude faster for an additive soft sphere mixture but yield a much more limited speed-up for the well-studied Kob–Andersen model. We discuss the current limitations of the method and suggest possible ways to improve it. [ABSTRACT FROM AUTHOR]

The deployment of the Industrial Internet of Things (IIoT) in smart container terminals provides a foundation for sensing and recording all operational processes. However, little effort has been devoted to integrating the massive data regarding interoperability challenges, thus limiting the value of data in advancing the intelligent evolution of ports. In this research, we propose a graph-based approach to organise operational records semantically, thereby facilitating data-driven decision-making in container terminals. We first construct a knowledge graph for operational processes in container terminals, employing a tailored procedure for the automatic conversion of operational records into triples. By utilising the graph information, we propose a novel method that integrates reinforcement learning (RL) with a mathematical solver for optimising scheduling problems. The quay crane scheduling problem (QCSP) is illustrated as an example to elaborate on the technical details. Based on a dataset from a real-world container terminal, numerical studies demonstrate the superiority of the proposed framework in terms of information retrieval efficiency and solution quality compared with the traditional data organisation approach. [ABSTRACT FROM AUTHOR]

We propose a scheme for achieving basic quantum gates using ultracold polar molecules in pendular states. The qubits are encoded in the YbF molecules trapped in an electric field with a certain gradient and coupled by the dipole–dipole interaction. The time-dependent control sequences consisting of multiple pulses are considered to interact with the pendular qubits. To achieve high-fidelity quantum gates, we map the control problem for the coupled molecular system into a Markov decision process and deal with it using the techniques of deep reinforcement learning (DRL). By training the agents over multiple episodes, the optimal control pulse sequences for the two-qubit gates of NOT, controlled NOT, and Hadamard are discovered with high fidelities. Moreover, the population dynamics of YbF molecules driven by the discovered gate sequences are analyzed in detail. Furthermore, by combining the optimal gate sequences, we successfully simulate the quantum circuit for entanglement. Our findings could offer new insights into efficiently controlling molecular systems for practical molecule-based quantum computing using DRL. [ABSTRACT FROM AUTHOR]

This paper studies a novel robotic warehousing system called the overhead robotic compact storage and retrieval system, which can free up the floor space occupation at a low cost. Bins, as basic storage containers, are stacked on top of each other to form a bin stack. Along overhead tracks, bin-picking robots transport bins between storage/retrieval positions and workstations with the aid of track-changing robots. Little research has been done to study operational policies and performance analysis for this new robotic compact warehousing system. We propose a nested queuing network model that considers two transportation resources and performs reinforcement learning using real data to improve the reshuffling efficiency. We find that reinforcement learning based reshuffling policy greatly reduces the reshuffling distance and saves computation time compared to existing policies. We find that the storage policy of stacks affects the optimal width/length ratio regardless of the system height. Interestingly, we obtain the number of robots that can stabilise the system to avoid an explosion of the order queue; two more robots than that number will produce relatively low throughput times. Compared to an AutoStore system, using our system reduces cost by 30% with a slight increase in throughput time. [ABSTRACT FROM AUTHOR]

Thermophotovoltaic (TPV) systems can be used to harvest thermal energy for thermoelectric conversion with much improved efficiency and power density compared with traditional photovoltaic systems. As the key component, selective emitters (SEs) can re-emit tailored thermal radiation for better matching with the absorption band of TPV cells. However, current designs of the SEs heavily rely on empirical design templates, particularly the metal-insulator-metal (MIM) structure, and lack of considering the overall performance of TPV systems and optimization efficiency. Here, we utilized a deep reinforcement learning (DRL) method to perform a comprehensive design of a 2D square-pattern metamaterial SE, with simultaneous optimization of material selections and structural parameters. In the DRL method, only the database of refractory materials with gradient refraction indexes needs to be prepared in advance, and the whole design roadmap will automatically output the SE with optimal Figure-of-Merit (FoM) efficiently. The optimal SE is composed of a novel material combination of TiO2, Si, and W substrate, with its thickness and structure precisely optimized. Its emissivity spectra match well with the external quantum efficiency curve of the GaSb cell. Consequently, the overall performance of TPV is significantly enhanced with an output power density of 5.78 W/cm2, an energy conversion efficiency of 38.26%, and a corresponding FoM of 2.21, surpassing most existing designs. The underlying physics of optimal SE is explained by the coupling effect of multiple resonance modes. This work advances the practical application potential of TPV systems and paves the way for addressing other multi-physics optimization problems and metamaterial designs. [ABSTRACT FROM AUTHOR]

Sequential optimization is one of the promising approaches in identifying the optimal candidate(s) (molecules, reactants, drugs, etc.) with desired properties (reaction yield, selectivity, efficacy, etc.) from a large set of potential candidates, while minimizing the number of experiments required. However, the high dimensionality of the feature space (e.g., molecular descriptors) makes it often difficult to utilize the relevant features during the process of updating the set of candidates to be examined. In this article, we developed a new sequential optimization algorithm for molecular problems based on reinforcement learning, multi-armed linear bandit framework, and online, dynamic feature selections in which relevant molecular descriptors are updated along with the experiments. We also designed a stopping condition aimed to guarantee the reliability of the chosen candidate from the dataset pool. The developed algorithm was examined by comparing with Bayesian optimization (BO), using two synthetic datasets and two real datasets in which one dataset includes hydration free energy of molecules and another one includes a free energy difference between enantiomer products in chemical reaction. We found that the dynamic feature selection in representing the desired properties along the experiments provides a better performance (e.g., time required to find the best candidate and stop the experiment) as the overall trend and that our multi-armed linear bandit approach with a dynamic feature selection scheme outperforms the standard BO with fixed feature variables. The comparison of our algorithm to BO with dynamic feature selection is also addressed. [ABSTRACT FROM AUTHOR]

The container pre-marshalling problem (CPMP) aims to minimise the number of reshuffling moves, ultimately achieving an optimised stacking arrangement in each bay based on the priority of containers during the non-loading phase. Given the sequential decision nature, we formulated the CPMP as a Markov decision process (MDP) model to account for the specific state and action of the reshuffling process. To address the challenge that the relocated container may trigger a chain effect on the subsequent reshuffling moves, this paper develops an improved policy-based Monte Carlo tree search (P-MCTS) to solve the CPMP, where eight composite reshuffling rules and modified upper confidence bounds are employed in the selection phases, and a well-designed heuristic algorithm is utilised in the simulation phases. Meanwhile, considering the effectiveness of reinforcement learning methods for solving the MDP model, an improved Q-learning is proposed as the compared method. Numerical results show that the P-MCTS outperforms all compared methods in scenarios where all containers have different priorities and scenarios where containers can share the same priority. [ABSTRACT FROM AUTHOR]

Photogrammetry systems are widely used in industrial manufacturing applications as an assistance measurement tool. Not only does it provide high-precision feedback for assembly process inspection and product quality assessment, but also it can improve the flexibility and robustness of manufacturing systems and production lines. However, with growing global competition and demands, companies are forced to enhance production efficiency, shorten production lifecycle and increase product variety by incorporating reconfigurable factory design that can meet challenging timeline and requirements. Although dynamic facility layout is widely investigated, the position selection for the photogrammetry system in dynamic manufacturing environment is usually overlooked. In this paper, dynamic layout of the V-STARS photogrammetry system is investigated and optimised in a digital-twin environment using deep reinforcement learning. The learning objectives are derived from the field of view (FoV) evaluation from point clouds 3D reconstruction, and collision detection from the digital twin simulated in Visual Components. The application feasibility of the proposed dynamic layout optimisation of the V-STARS photogrammetry system is verified with a real world industrial application. [ABSTRACT FROM AUTHOR]

Dynamic task scheduling problem in cloud manufacturing (CMfg) is always challenging because of changing manufacturing requirements and services. To make instant decisions for task requirements, deep reinforcement learning-based (DRL-based) methods have been broadly applied to learn the scheduling policies of service providers. However, the current DRL-based scheduling methods struggle to fine-tune a pre-trained policy effectively. The resulting training from scratch takes more time and may easily overfit the environment. Additionally, most DRL-based methods with uneven action distribution and inefficient output masks largely reduce the training efficiency, thus degrading the solution quality. To this end, this paper proposes an improved DRL-based approach for dynamic task scheduling in CMfg. First, the paper uncovers the causes behind the inadequate fine-tuning ability and low training efficiency observed in existing DRL-based scheduling methods. Subsequently, a novel approach is proposed to address these issues by updating the scheduling policy while considering the distribution distance between the pre-training dataset and the in-training policy. Uncertainty weights are introduced to the loss function, and the output mask is extended to the updating procedures. Numerical experiments on thirty actual scheduling instances validate that the solution quality and generalization of the proposed approach surpass other DRL-based methods at most by 32.8% and 28.6%, respectively. Additionally, our method can effectively fine-tune a pre-trained scheduling policy, resulting in an average reward increase of up to 23.8%. [ABSTRACT FROM AUTHOR]

As a new production pattern, the hybrid seru system (HSS) originated from the actual production scenario. In the HSS, the implementation of the worker transfer strategy can further enhance the system's flexibility but is rarely studied at present. In this paper, we develop a reinforcement learning driven two-stage evolutionary algorithm (RL-TEA) to address the hybrid seru system scheduling problem with worker transfer (HSSSP-WT). To conquer this complex problem, the HSSSP-WT is divided into worker assignment-related subproblems (WS) and batch scheduling-related subproblems (BS) according to the problem characteristics. To effectively solve the subproblems, a probability modelbased exploration and a lower bound-guided heuristic are presented for the WS, and a greedy search is designed for the BS. Meanwhile, to improve search efficiency and effectiveness, a knowledgebased selection mechanism is proposed to determine which subproblem group to optimise in each generation by fusing a reinforcement learning technique and a lower bound filtering strategy. Moreover, an elite enhancement strategy inspired by the problem property is designed to improve the solution quality. Experimental results demonstrate the effectiveness of the worker transfer strategy and the superior performance of the RL-TEA compared with the state-of-the-art algorithms in solving the HSSSP-WT. [ABSTRACT FROM AUTHOR]

Herein, we present a real-time multi-agent deep reinforcement learning model as a disassembly planning framework for human-robot collaboration. This disassembly plan optimises sequences to minimise operation time and the disassembling costs of end-of-life (EoL) products. Combining different data-driven decision-making tools, the plan aims to handle the complexities and uncertainties of disassembly tasks. Based on the physical features and geometric limitations of EoL product components, we calculate product disassembly difficulty scores. Subsequently, the deep reinforcement learning model integrates these scores into planning process. The model allocates tasks in real time according to the online conditions of the human operator, cobot, and product, enabling the model to cope with uncertainties that may change the process routine. We also present different scenarios wherein a cobot collaborates with human operators with different skill levels. To evaluate model performance, we compare it with baseline models in terms of the convergence time and incorporated disassembly features. The analysis indicates that our model converges three times faster than a baseline model applied to the same case study. Moreover, our model includes more features of the disassembly problem in its decision-making process than any other baseline model. [ABSTRACT FROM AUTHOR]

Unlike mass production manufacturing systems, where configurations are rarely changed after the initial design, reconfigurable cyber-physical systems (RCPMS) self-change their structures throughout missions and thus self-adjust production in response to demand requirements. Accordingly, such a paradigm requires enhancing selective maintenance strategy to optimise scheduling maintenance actions, selecting configuration layouts for capacity and product family changes, and achieving maintenance cost reduction and reliability maximisation. This paper is the first to propose a robust model for a selective maintenance problem with imperfect repairs in the RCPMS context. The model also integrates uncertainties originating from the imperfect observations of components' health status. The model's objectives are to maximise the expected reliability and minimise the variance and maintenance cost under maintenance resource constraints. Moreover, we propose a new deep reinforcement learning framework for solving the resulting multi-objective and combinatorial optimisation problem. In addition, we use decision values to enhance the scalarisation process by permitting the priorities of specific objectives to be adjusted after the learning process. Furthermore, we employ Analytical Hierarchy Process to adjust the static priorities with respect to the objective functions and the actual learning context. Finally, broad experiments are conducted to highlight the performance of the proposed model and resolution framework. [ABSTRACT FROM AUTHOR]

In contrast to the parent compound YCo12B6, the Fe-doped YCo12−xFexB6 (1.5 ≤ x ≤ 2.5) alloys exhibit an antiferromagnetic ground state. It is further revealed that the antiferromagnetic state gets transformed into a forced ferromagnetic state by way of a magnetic-field-induced metamagnetic transition. The results demonstrate a progressive reinforcement of the antiferromagnetic interactions upon increasing Fe content along the YCo12−xFexB6 series of compounds. The magnetic phase diagram of each (x = 1.5, 2, and 2.5) composition is determined by combining isothermal and isofield magnetization measurements. The composition and temperature dependencies of the critical field are derived and discussed. Whereas for x = 2 and 2.5, the magnetic state changes from antiferromagnetic (AFM) to ferromagnetic (FM) and then to the paramagnetic (PM) state upon heating, exhibiting three distinct temperature regions, a multicritical point of magnetic origin is proposed for the x = 1.5 compound at a temperature of about 117 K. The ordering temperature is found to decrease upon increasing Fe concentration, while the critical field of the AFM-FM metamagnetic phase transition shows the opposite trend. The critical transition field diminishes upon increasing temperature. [ABSTRACT FROM AUTHOR]

In nowadays production, fluctuations in demand, shortening product life-cycles, and highly configurable products require an adaptive and robust control approach to maintain competitiveness. This approach must not only optimise desired production objectives but also cope with unforeseen machine failures, rush orders, and changes in short-term demand. Previous control approaches were often implemented using a single operations layer and a standalone deep learning approach, which may not adequately address the complex organisational demands of modern manufacturing systems. To address this challenge, we propose a hyper-heuristics control model within a semi-heterarchical production system, in which multiple manufacturing and distribution agents are spread across pre-defined modules. The agents employ a deep reinforcement learning algorithm to learn a policy for selecting low-level heuristics in a situation-specific manner, thereby leveraging system performance and adaptability. We tested our approach in simulation and transferred it to a hybrid production environment. By that, we were able to demonstrate its multi-objective optimisation capabilities compared to conventional approaches in terms of mean throughput time, tardiness, and processing of prioritised orders in a multi-layered production system. The modular design is promising in reducing the overall system complexity and facilitates a quick and seamless integration into other scenarios. [ABSTRACT FROM AUTHOR]

Generative models in the field of artificial intelligence and their applications and deployment have demonstrated their great strength in the past few years. Of the vast spectrum of generative models, diffusion probabilistic models have proven to be particularly powerful and productive, transforming notions such as text-to-image and text-to-video generation from ideas into practical applications. In our previous works, we proposed a thermal metamaterial-based periodic interparticle interaction mechanism for heat management, with a specific application in thermal transparency. To address the challenging problems associated with the inverse design of thermal metamaterial structures, we employed an autoencoder-based machine learning approach and a reinforcement learning-based approach successfully. In this work, we demonstrate that our particular problems with the inverse design of thermal metamaterial-based periodic lattices for the realization of thermal transparency can also be reframed and efficiently solved by training a generative diffusion probabilistic model that can generate the design parameters corresponding to the desired response. Furthermore, we show that for a specific response, multiple sets of design parameters can be obtained by simply performing multiple inferences with the generative diffusion probabilistic model, enabling us to select the ones that can be more economical to fabricate and implement. Our work is among the first to use a diffusion model for the inverse design of thermal metamaterial-based structures and demonstrates the effectiveness of generating low-dimensional design parameters through a diffusion model. [ABSTRACT FROM AUTHOR]

Finding the optimal treatment strategy to accelerate wound healing is of utmost importance, but it presents a formidable challenge owing to the intrinsic nonlinear nature of the process. We propose an adaptive closed-loop control framework that incorporates deep learning, optimal control and reinforcement learning to accelerate wound healing. By adaptively learning a linear representation of nonlinear wound healing dynamics using deep learning and interactively training a deep reinforcement learning agent for tracking the optimal signal derived from this representation without the need for intricate mathematical modelling, our approach has not only successfully reduced the wound healing time by 45.56% compared to the one without any treatment, but also demonstrates the advantages of offering a safer and more economical treatment strategy. The proposed methodology showcases a significant potential for expediting wound healing by effectively integrating perception, predictive modelling and optimal adaptive control, eliminating the need for intricate mathematical models.

The utility of a quantum computer is highly dependent on the ability to reliably perform accurate quantum logic operations. For finding optimal control solutions, it is of particular interest to explore model-free approaches, since their quality is not constrained by the limited accuracy of theoretical models for the quantum processor—in contrast to many established gate implementation strategies. In this work, we utilize a continuous control reinforcement learning algorithm to design entangling two-qubit gates for superconducting qubits; specifically, our agent constructs cross-resonance and CNOT gates without any prior information about the physical system. Using a simulated environment of fixed-frequency fixed-coupling transmon qubits, we demonstrate the capability to generate novel pulse sequences that outperform the standard cross-resonance gates in both fidelity and gate duration, while maintaining a comparable susceptibility to stochastic unitary noise. We further showcase an augmentation in training and input information that allows our agent to adapt its pulse design abilities to drifting hardware characteristics, importantly, with little to no additional optimization. Our results exhibit clearly the advantages of unbiased adaptive-feedback learning-based optimization methods for transmon gate design.

This paper addresses the problem of stochastic jointly scheduling of resumable jobs and preventive maintenance on a single machine, subject to random breakdowns, to minimise the earliness-tardiness cost. The main objective is to investigate using trending machine learning-based methods compared to stochastic optimisation approaches. We propose two different methods from both fields as we solve the same problem firstly with a stochastic dynamic programming model in an approximation way, then with an attention-based deep reinforcement learning model. We conduct a detailed experimental study according to solution quality, run time, and robustness to analyse their performances compared to those of an existing approach in the literature as a baseline. Both algorithms outperform the baseline. Moreover, the machine learning-based algorithm outperforms the stochastic dynamic programming-based heuristic as we report up to 30.5% saving in total cost, a reduction of computational time from 67 min to less than $ 1s $ 1 s on big instances, and a better robustness. These facts highlight clearly its potential for solving such problems. [ABSTRACT FROM AUTHOR]

This paper proposes a novel intelligent scheduling method based on deep reinforcement learning (DRL) to solve the multi-objective steelmaking-continuous casting (SCC) scheduling problem, under time-of-use (TOU) tariffs for the first time. The intelligent scheduling system architecture is designed, and a mathematical model is established to minimise the total sojourn time and electricity cost. To effectively reduce production costs by avoiding peak periods of electricity consumption, the 'start time' of the system is generated based on the Markov Decision Process (MDP), and heuristic scheduling rules related to power cost are used as the action space, with corresponding reward functions designed according to the characteristics of these two objectives. To satisfy the continuous casting which is a particular SCC constraint, a backward strategy is developed. Additionally, a branching duelling double deep Q-network (BD3QN) is adapted to guide action selection and avoid blind search in the iteration process, and then applied to real-time scheduling. Numerical experiments demonstrate that the proposed method outperforms comparison algorithms in terms of solution quality and CPU times by a large margin. [ABSTRACT FROM AUTHOR]

Cloud computing is widely applied in modern industrial areas due to its technological advancement, cost reduction, and applicability. Packets (tasks) belonging to different applications (agents) compete to share the common cloud resource through a series of edge nodes (processors) in pursuit of fast transmission. This paper abstracts the cloud computing system as a multi-agent flow-shop scheduling (MAFS) problem. The objective is to minimise the total completion time of several agents with the restriction that the maximum lateness cannot exceed a given bound. Given the complexity of the considered problem, a branch and bound algorithm combined with several pruning rules and lower bounds is proposed to obtain optimal solutions. Furthermore, the considered problem is generalised to a bi-scenario version, and a bi-population cooperative co-evolutionary (BCCE) algorithm is proposed to solve it. A reinforcement learning-based method is presented to generate the initial population. Several problem-specific intensification strategies are constructed to explore promising solutions. Comprehensive experiments verified the effectiveness of the proposed algorithms. The industrial data from the China Earthquake Network Centre further confirmed the superiority of the BCCE algorithm. Overall, the MAFS model and the proposed algorithms effectively enhance the user experience and reasonably guarantee revenue. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning (DRL)-based control shows enhanced performance in the management of integrated energy systems when compared with Rule-Based Controllers (RBCs), but it still lacks scalability and generalisation due to the necessity of using tailored models for the training process. Transfer Learning (TL) is a potential solution to address this limitation. However, existing TL applications in building control have been mostly tested among buildings with similar features, not addressing the need to scale up advanced control in real-world scenarios with diverse energy systems. This paper assesses the performance of an online heterogeneous TL strategy, comparing it with RBC and offline and online DRL controllers in a simulation setup using EnergyPlus and Python. The study tests the transfer in both transductive and inductive settings of a DRL policy designed to manage a chiller coupled with a Thermal Energy Storage (TES). The control policy is pre-trained on a source building and transferred to various target buildings characterised by an integrated energy system including photovoltaic and battery energy storage systems, different building envelope features, occupancy schedule and boundary conditions (e.g., weather and price signal). The TL approach incorporates model slicing, imitation learning and fine-tuning to handle diverse state spaces and reward functions between source and target buildings. Results show that the proposed methodology leads to a reduction of 10% in electricity cost and between 10% and 40% in the mean value of the daily average temperature violation rate compared to RBC and online DRL controllers. Moreover, online TL maximises self-sufficiency and self-consumption by 9% and 11% with respect to RBC. Conversely, online TL achieves worse performance compared to offline DRL in either transductive or inductive settings. However, offline Deep Reinforcement Learning (DRL) agents should be trained at least for 15 episodes to reach the same level of performance as the online TL. Therefore, the proposed online TL methodology is effective, completely model-free and it can be directly implemented in real buildings with satisfying performance.