Search

Distributed, consensus-based algorithms have emerged as a promising approach for the coordination of distributed energy resources (DER) due to their communication, computation, privacy and reliability advantages over centralized approaches. However, state-of-the-art consensus-based algorithms address the DER coordination problem in independent time periods and therefore are inherently unable to capture the time-coupling operating characteristics of flexible demand (FD) and energy storage (ES) resources. This paper demonstrates that state-of-the-art algorithms fail to converge when these time-coupling characteristics are considered. In order to address this fundamental limitation, a novel consensus-based algorithm is proposed which includes additional consensus variables. These variables express relative maximum power limits imposed on the FD and ES resources which effectively mitigate the concentration of the FD and ES responses at the same time periods and steer the consensual outcome to a feasible and optimal solution. The convergence and optimality of the proposed algorithm are theoretically proven while case studies numerically demonstrate its convergence, optimality, robustness to initialization and information loss, and plug-and-play adaptability.

Peer-to-peer (P2P) energy trading has emerged as a promising market paradigm toward maximizing the value of distributed energy resources (DERs) for electricity prosumers, by enabling direct energy trading among them. However, state-of-the-art P2P mechanisms either fail to adequately incentivize prosumers to participate, prevent prosumers from accessing the highest achievable monetary benefits, or suffer severely from the curse of dimensionality. This article proposes two computationally efficient mechanisms to construct a stable grand coalition of prosumers participating in P2P trading, founded on cooperative game-theoretic principles. The first one involves a benefit distribution scheme inspired by the core tâtonnement process while the second involves a novel pricing mechanism based on the solution of a single linear program. The performance of the proposed mechanisms is validated against state-of-the-art mechanisms through numerous case studies using real-world data. The results demonstrate that the proposed mechanisms exhibit superior computational performance than the nucleolus and are superior to the rest of the examined mechanisms in incentivizing prosumers to remain in the grand coalition.

The effective pricing of electric vehicle (EV) charging by aggregators constitutes a key problem toward the realization of the significant EV flexibility potential in deregulated electricity systems and has been addressed by previous work through bi-level optimization formulations. However, the solution approach adopted in previous work cannot capture the discrete nature of the EV charging/discharging levels. Although reinforcement learning (RL) can tackle this challenge, state-of-the-art RL methods require discretization of state and/or action spaces and thus exhibit limitations in terms of solution optimality and computational requirements. This article proposes a novel deep reinforcement learning (DRL) method to solve the examined EV pricing problem, combining deep deterministic policy gradient (DDPG) principles with a prioritized experience replay (PER) strategy and setting up the problem in multi-dimensional continuous state and action spaces. Case studies demonstrate that the proposed method outperforms state-of-the-art RL methods in terms of both solution optimality and computational requirements and comprehensively analyze the economic impacts of smart-charging and vehicle-to-grid (V2G) flexibility on both aggregators and EV owners.

Multi-energy systems (MES) are attracting increasing attention driven by its potential to offer significant flexibility in future smart grids. At the residential level, the roll-out of smart meters and rapid deployment of smart energy devices call for autonomous multi-energy management systems which can exploit real-time information to optimally schedule the usage of different devices with the aim of minimizing end-users’ energy costs. This paper proposes a novel real-time autonomous energy management strategy for a residential MES using a model-free deep reinforcement learning (DRL) based approach, combining state-of-the-art deep deterministic policy gradient (DDPG) method with an innovative prioritized experience replay strategy. This approach is tailored to align with the nature of the problem by posing it in multi-dimensional continuous state and action spaces, facilitating more cost-effective control strategies to be devised. The superior performance of the proposed approach in reducing end-user’s energy cost while coping with the MES uncertainties is demonstrated by comparing it against state-of-the-art DRL methods as well as conventional stochastic programming and robust optimization methods in numerous case studies in a real-world scenario.

Hydropower is a clean, low-carbon renewable energy source with the merits of mature technology and flexible operation. Moreover, hydropower plants provide comprehensive utilization functions in many areas, including flood control, shipping, and emission reduction. In the current contradictive situation between power supply and demand, China's hydropower grid pricing mechanism does not reflect the positive externality of hydropower, which is not conducive to fair competition between hydropower and other power sources. This study proposed the concept of the positive external electricity price (PEEP) and a pricing mechanism that considered, for the first time, the values of hydropower positive externality factors by using the positive externality theory. A representative quasi-public welfare power plant of the Sichuan power grid was used to empirically evaluate the model. The pricing mechanism of reservoir power plants proposed in this paper considered the power grid benefits, flood control benefits, shipping benefits, and emission reduction benefits of hydropower, and allowed for the internalization of the positive externality. The hydropower grid prices based on our recommendations did not cause a dramatic impact on the local power grid; furthermore, the model helped promote the social responsibility of power companies through the pricing guidance for improving economic benefits by increasing positive externalities.

The investigation of the effects of demand flexibility on the pricing strategies and the profits of electricity retailers has recently emerged as a highly interesting research area. However, the state-of-the-art, bi-level optimisation modelling approach makes the unrealistic assumption that retailers treat wholesale market prices as exogenous, fixed parameters. This study proposes a tri-level optimisation model, which drops this assumption and represents the wholesale market clearing process endogenously, thus capturing the realistic implications of a retailer's pricing strategies and the resulting demand response on the wholesale market prices. The scope of the examined case studies is three-fold. First of all, they demonstrate the interactions between the retailer, the flexible consumers and the wholesale market and analyse the fundamental effects of the consumers’ time-shifting flexibility on the retailer's revenue from the consumers, its cost in the wholesale market and its overall profit. Furthermore, they analyse how these effects of demand flexibility depend on the retailer's relative size in the market and the strictness of the regulatory framework. Finally, they highlight the added value of the proposed tri-level model by comparing its outcomes against the state-of-the-art bi-level modelling approach.

Bi-level optimization constitutes the most popular mathematical methodology for modeling the deregulated electricity market. However, state-of-the-art models neglect the physical non-convex operating characteristics of market participants, due to their inherent inability to capture binary decision variables in their representation of the market clearing process, rendering them problematic in modeling markets with complex bidding and unit commitment (UC) clearing mechanisms. This paper addresses this fundamental limitation by proposing a novel modeling approach enabling incorporation of these non-convexities into bi-level optimization market models, which is based on the relaxation and primal-dual reformulation of the original, non-convex lower level problem and the penalization of the associated duality gap. Case studies demonstrate the ability of the proposed approach to closely approximate the market clearing solution of the actual UC clearing algorithm and devise more profitable bidding decisions for strategic producers than the state-of-the-art bi-level optimization approach, and reveal the potential of strategic behavior in terms of misreporting non-convex operating characteristics.

Demand Response (DR) is one of the most cost-effective solutions for providing flexibility to power systems. The extensive deployment of DR trials and the roll-out of smart meters enable the quantification of consumer responsiveness to price signals via baseline estimation. The traditional deterministic baseline estimation approach can provide only a single value without consideration of uncertainty. This paper proposes a novel probabilistic baseline estimation framework that consists of a daily load profile pool construction stage, a deep learning-based clustering stage, an optimal cluster selection stage, and a quantile regression forests model construction stage. In particular, the concept of a daily load profile pool is introduced, and a deep-learning-based clustering approach is employed to handle a large number of daily patterns to further improve the baseline estimation performance. Case studies have been conducted on fine-grained smart meter data collected from a real dynamic time-of-use (dTOU) tariffs trial of the Low Carbon London (LCL) project. The superior performance of the proposed method is demonstrated based on a series of evaluation metrics regarding both deterministic and probabilistic estimation results.

Decarbonization of the electricity system will require significant and continued investment in low-carbon energy sources and electrification of the heat and transport sectors. With diminishing output and shorter operating hours of conventional large-scale fossil-fueled generators, there is a growing need and opportunity for distributed energy resources (DERs) to contribute to the provision of system balancing and security services and support a cost-effective transition to a lower-carbon energy system. Emerging smart technologies will open new opportunities for millions of users to participate actively in the trading of electricity and various ancillary services through alternative mechanisms, such as dynamic pricing, local energy markets, security markets, and peer-to-peer (P2P) trading.

With the roll-out of smart meters and the increasing prevalence of distributed energy resources (DERs) at the residential level, end-users rely on home energy management systems (HEMSs) that can harness real-time data and employ artificial intelligence techniques to optimally manage the operation of different DERs, which are targeted toward minimizing the end-user’s energy bill. In this respect, the performance of the conventional model-based demand response (DR) management approach may deteriorate due to the inaccuracy of the employed DER operating models and the probabilistic modeling of uncertain parameters. To overcome the above drawbacks, this paper develops a novel real-time DR management strategy for a residential household based on the twin delayed deep deterministic policy gradient (TD3) learning approach. This approach is model-free, and thus does not rely on knowledge of the distribution of uncertainties or the operating models and parameters of the DERs. It also enables learning of neural-network-based and fine-grained DR management policies in a multi-dimensional action space by exploiting high-dimensional sensory data that encapsulate the uncertainties associated with the renewable generation, appliances’ operating states, utility prices, and outdoor temperature. The proposed method is applied to the energy management problem for a household with a portfolio of the most prominent types of DERs. Case studies involving a real-world scenario are used to validate the superior performance of the proposed method in reducing the household’s energy costs while coping with the multi-source uncertainties through comprehensive comparisons with the state-of-the-art deep reinforcement learning (DRL) methods.

Local energy markets (LEM) have recently attracted great interest as they enable effective coordination of small-scale distributed energy resources (DER) at the customer side, and avoidance of distribution network reinforcements. However, the introduction of LEM has also significant implications on the strategic interactions between the customers and incumbent electricity retailers. This paper explores for the first time these interactions by proposing a novel multi-period bi-level optimization model, which captures the pricing decisions of a strategic retailer in the upper level (UL) and the response of both independent customers and the LEM (both including flexible consumers, micro-generators and energy storages) in the lower level (LL). Since the LL problem representing the LEM is non-convex, a new analytical approach is employed for solving the developed bi-level problem. The examined case studies demonstrate that the introduction of an LEM reduces the customers’ energy dependency on the retailer and limits the retailer’s strategic potential of exploiting the customers through large differentials between buy and sell prices. As a result, the profit of the retailer is significantly reduced while the customers, primarily the LEM participants and to a lower extent non-participating customers, achieve significant economic benefits.

Bi-level optimization and reinforcement learning (RL) constitute the state-of-the-art frameworks for modeling strategic bidding decisions in deregulated electricity markets. However, the former neglects the market participants’ physical non-convex operating characteristics, while conventional RL methods require discretization of state and / or action spaces and thus suffer from the curse of dimensionality. This paper proposes a novel deep reinforcement learning (DRL) based methodology, combining a deep deterministic policy gradient (DDPG) method with a prioritized experience replay (PER) strategy. This approach sets up the problem in multidimensional continuous state and action spaces, enabling market participants to receive accurate feedback regarding the impact of their bidding decisions on the market clearing outcome, and devise more profitable bidding decisions by exploiting the entire action domain, also accounting for the effect of non-convex operating characteristics. Case studies demonstrate that the proposed methodology achieves a significantly higher profit than the alternative state-of-the-art methods, and exhibits a more favorable computational performance than benchmark RL methods due to the employment of the PER strategy.

Bi-level optimization and reinforcement learning (RL) constitute the state-of-the-art frameworks for modeling strategic bidding decisions in deregulated electricity markets. However, the former neglects the market participants' physical non-convex operating characteristics, while conventional RL methods require discretization of state and / or action spaces and thus suffer from the curse of dimensionality. This paper proposes a novel deep reinforcement learning (DRL) based methodology, combining a deep deterministic policy gradient (DDPG) method with a prioritized experience replay (PER) strategy. This approach sets up the problem in multi-dimensional continuous state and action spaces, enabling market participants to receive accurate feedback regarding the impact of their bidding decisions on the market clearing outcome, and devise more profitable bidding decisions by exploiting the entire action domain, also accounting for the effect of non-convex operating characteristics. Case studies demonstrate that the proposed methodology achieves a significantly higher profit than the alternative state-of-the-art methods, and exhibits a more favourable computational performance than benchmark RL methods due to the employment of the PER strategy.

Previous work on the role of the demand side in imperfect electricity markets has demonstrated that its self-price elasticity reduces electricity producers' ability to exercise market power. However, the concept of self-price elasticity cannot accurately capture consumers' flexibility, as the latter mainly involves shifting of loads' operation in time. This paper provides for the first time theoretical and quantitative analysis of the beneficial impact of demand shifting (DS) in mitigating market power by the generation side. Quantitative analysis is supported by a multiperiod equilibrium programming model of the imperfect electricity market, accounting for the time-coupling operational constraints of DS as well as network constraints. The decision making process of each strategic producer is modeled through a bi-level optimization problem, which is solved after converting it to a Mathematical Program with Equilibrium Constraints (MPEC) and linearizing the latter through suitable techniques. The oligopolistic market equilibria resulting from the interaction of multiple independent producers are determined by employing an iterative diagonalization method. Case studies on a test market reflecting the general generation and demand characteristics of the GB system quantitatively demonstrate the benefits of DS in mitigating producers' market power, by employing relevant indexes from the literature.

The increasing penetration of small-scale distributed energy resources (DER) has the potential to support cost-efficient energy balancing in emerging electricity systems, but is also fundamentally affecting the conventional operation paradigm of the latter. In this context, innovative market mechanisms need to be devised to better coordinate and provide incentives for DER to utilize their flexibility. Peer-to-Peer (P2P) energy trading has emerged as an alternative approach to facilitate direct trading between consumers and prosumers interacting in an energy collective and fosters more efficient local demand–supply balancing. While previous research has primarily focused on the technical and economic benefits of P2P trading, little effort has been made towards the incorporation of prosumers’ heterogeneous characteristics in the P2P trading problem. Here, we address this research gap by classifying the participating prosumers into multiple clusters with regard to their portfolio of DER, and analyzing their trading decisions in a simulated P2P trading platform. The latter employs the mid-market rate (MMR) local pricing mechanism to enable energy trading among prosumers and penalizes the contribution to the system demand peak of each prosumer. We formulate the P2P trading problem as a multi-agent coordination problem and propose a novel multi-agent deep reinforcement learning (MADRL) method to address it. The proposed method is founded on the combination of the multi-agent deep deterministic policy gradient (MADDPG) algorithm and the technique of parameter sharing (PS), which not only enables accelerating the training speed by sharing experiences and learned policies between all agents in each cluster, but also sustains the policies’ diversity between multiple clusters. To address the non-stationarity and computational complexity of MADRL as well as persevering the privacy of prosumers, the P2P trading platform acts as a trusted third party which augments the market collective trading information to help training of prosumer agents. Experiments with a large-scale real-world data-set involving 300 residential households demonstrate that the proposed MADRL method exhibits a strong generalization capability in the test data-set and outperforms the state-of-the-art MADRL methods with regard to the system operation cost, demand peak as well as computational time.

Previous work has analyzed the renewable generation hosting capacity of electricity systems and the relevant value of flexible demand from a techno-economic perspective, considering the balancing and network challenges associated with a large-scale integration of renewables. This paper investigates these issues from a market perspective, considering the challenges of low energy prices and renewables’ investment cost recovery. In this context, a multi-period bi-level optimization model is developed, where the upper level maximizes the renewable generation capacity subject to the long-term profitability constraint and the lower level represents the market clearing process, accounting for the time-coupling operational constraints of flexible demand. This bi-level problem is solved after converting it to a single-level mixed-integer linear problem (MILP). Case studies validate the proposed model and demonstrate that demand flexibility increases the maximum renewable generation capacity that can be integrated in the system without violating its profitability constraint.

Previous work has analyzed the role of energy storage (ES) on generation investment planning through centralised cost-minimization models which are inherited from the era of regulated electricity utilities. This paper investigates this issue in the context of the deregulated market environment by proposing a new strategic generation investment planning model. The decision making of a strategic generation company is modeled through a multi-period bi-level optimization problem, where the upper level determines the profit-maximizing investment decisions of the generation company and the lower level represents the market clearing process, accounting for the time-coupling operational characteristics of ES. This bi-level problem is solved after converting it to a single-level mixed-integer linear problem (MILP). Case studies demonstrate that the introduction of ES reduces the total generation capacity investment and enhances investments in “must-run” base-load generation over flexible peaking generation, yielding significant system cost savings.

The envisaged decarbonisation of electricity systems has attracted significant interest around the role and value of energy storage systems (ESSs). In the deregulated electricity market, there is a need to investigate the complex impacts of ESSs, considering the potential exercise of market power by strategic players. This study aims at comprehensively analysing the impacts of both price-taking and price-making storage behaviours on energy market efficiency, corresponding to potential settings with small and large storage players, respectively. In order to achieve this and in contrast to previous papers, this work develops a multi-period equilibrium programming market model to determine market equilibrium stemming from the interactions of independent strategic producers and ESSs, while capturing the time-coupling operational constraints of ESSs as well as network constraints. The results of case studies on a test market capturing the general conditions of the GB electricity system demonstrate that the introduction of ESSs mitigates market power exercise and improves market efficiency, with this beneficial impact being higher when ESSs act as price takers. When the electricity network is congested, the location of ESSs also affects the market outcome, with their beneficial impact on market efficiency being higher when they are located in higher-priced areas.

Load balancing issues in distribution networks have emerged alongside the large-scale deployment of distributed renewable generation sources. In light of this challenge, peer-to-peer (P2P) energy trading constitutes a promising approach for delivering secure and economic supply-demand balance when faced with variable load and intermittent renewable generation through matching energy demand and supply locally. However, state-of-the-art mechanisms for governing P2P energy trading either fail to suitably incentivize prosumers to participate in P2P trading or suffer severely from the curse of dimensionality with their computational complexity increase exponentially with the number of prosumers. In this paper, a P2P energy trading mechanism based on cooperative game theory is proposed to establish a grand energy coalition of prosumers and a computationally efficient pricing algorithm is developed to suitably incentivize prosumers for their sustainable participation in the grand coalition. The performance of the proposed algorithm is demonstrated by comparing it to state-of-the-art mechanisms through numerous case studies in a real-world scenario. The superior computational performance of the proposed algorithm is also validated.

Previous work has demonstrated the significant benefits of demand flexibility for low-carbon electricity systems. However, given the deregulation of the electricity industry, this flexibility also affects the business case of retailers, an aspect which has not yet been thoroughly investigated. This paper proposes a new multi-period bi-level optimization model capturing the interactions between a retailer's offered tariffs to the served consumers, the consumers' flexibility and the wholesale market in an integrated fashion. This bi-level problem is efficiently solved after converting it to a Mathematical Program with Equilibrium Constraints (MPEC), and subsequently to a Mixed-Integer Linear Problem (MILP). Case studies demonstrate that the time-shifting flexibility of the consumers reduces the retailer's revenue, since consumers can respond more effectively to strategic retail pricing patterns, but also reduces the retailer's cost, since it leads to flatter wholesale demand and price profiles, an effect which is not captured by previous work. Finally, the results indicate that the impact of this flexibility on the retailer's business case depends on the level of competition with other retailers in the market.

The envisaged decarbonization of electricity systems has attracted significant inte rest around the role and value of demand flexibility . However, the impact of this flexibility on generation investments in the deregulated electricity industry setting remains a largely unexplored area, since previous relevant work neglects the time - coupli ng nature of demand shifting potentials. This paper addresses this challenge by proposing a strategic generation investment planning model expressing the decision making process of a self - interested generation company and accounting for the time - coupling o perational characteristics of demand flexibility. This model is formulated as a multi - period bi - level optimization problem, which is solved after converting it to a Mathematical Program with Equilibrium Constraints (MPEC). Case studies with the proposed mo del demonstrate that demand flexibility reduces the total generation capacity investment, enhances investments in baseload generation and yield s significant economic benefits in terms of total system costs and demand payments.

Previous work has demonstrated that the price elasticity of the demand side reduces electricity producers' ability to exercise market power. However, price elasticity cannot capture alone consumers' flexibility, as the latter mainly involves shifting of loads' operation in time. This paper provides for the first time qualitative and quantitative analysis of the value of demand shifting in mitigating market power by the generation side. An equilibrium programming model of the oligopolistic market setting is developed, taking into account the inter-temporal characteristics of demand shifting. The decision making process of each strategic producer is modelled through a bi-level optimization problem, which is solved after transforming it to a Mathematical Program with Equilibrium Constraints (MPEC). The market equilibria resulting from the interaction of multiple independent producers are determined by employing an iterative diagonalization method. Case studies on a test market with day-ahead horizon and hourly resolution quantitatively demonstrate the benefits of demand shifting in limiting generation market power, by employing relevant indexes from the literature.

Although previous work has demonstrated the ability of large energy storage (ES) units to exercise market power by withholding their capacity, it has adopted modeling approaches exhibiting certain limitations and has not analyzed the dependency of the extent of exercised market power on ES operating properties. In this paper, the decision making process of strategic ES is modeled through a bi-level optimization problem; the upper level determines the optimal extent of capacity withholding at different time periods, maximizing the ES profit, while the lower level represents endogenously the market clearing process. This problem is solved after converting it to a Mathematical Program with Equilibrium Constraints (MPEC) and linearizing the latter through suitable techniques. Case studies on a test market quantitatively analyze the extent of capacity withholding and its impact on ES profit and social welfare for different scenarios regarding the power and energy capacity of ES.

Although recent studies have investigated the impacts of energy storage on various aspects of power system operation and planning, its role in imperfect electricity markets has not been explored yet. This paper provides for the first time theoretical and quantitative evidence of the beneficial impact of energy storage in limiting market power by generation companies. Quantitative analysis is supported by a bi-level optimization model of the imperfect electricity market setting, accounting for the time-coupling operational constraints of energy storage. This bi-level problem is solved after converting it to a Mathematical Program with Equilibrium Constraints (MPEC). Case studies are carried out on a test market with day-ahead horizon and hourly resolution.

The use of hydropower has expanded in recent years in southwest China, playing a key role in the success of China's West-to-East energy strategy. Although installed hydropower capacity has reached a record high, the problem of abandoned water has intensified and has been gradually highlighted by researchers. Abandoned water has resulted in a significant waste of carbon-free and inexpensive renewable energy, which is not conducive to China's goals to reduce emissions to save energy. This waste has seriously affected the development of China's hydropower industry. Therefore, now is the time to analyze the main reasons for the abandoned water problem and to identify the crux of the problem. In this paper, we collected the actual operation data of all the power stations in the case area and established the mathematical model according to the quantity of abandoned water as well as the actual and the theoretical output curve of the power station to quantify the abandoned water into energy loss. We then classified the abandoned water according to three groups of causes, namely, generation-side, line-side, and market-side causes. In 2016, the total energy loss of abandoned water of Sichuan Province was 28.73 × 109 kW h, of which the generation-, line-, and market-side causes account for 7.76%, 11.68%, and 80.56% of the energy loss, respectively. Therefore, the market-side causes are the main cause for the abandoned water problem. Such market-side causes primarily include the imbalance between electricity supply and demand, the provincial barriers of the local government system and its mechanisms, and the hysteresis congestion of the transmission lines. Seizing the favorable opportunity presented by new power system reforms in China, this study proposes several market-oriented measures to improve hydropower consumption capacity.The use of hydropower has expanded in recent years in southwest China, playing a key role in the success of China's West-to-East energy strategy. Although installed hydropower capacity has reached a record high, the problem of abandoned water has intensified and has been gradually highlighted by researchers. Abandoned water has resulted in a significant waste of carbon-free and inexpensive renewable energy, which is not conducive to China's goals to reduce emissions to save energy. This waste has seriously affected the development of China's hydropower industry. Therefore, now is the time to analyze the main reasons for the abandoned water problem and to identify the crux of the problem. In this paper, we collected the actual operation data of all the power stations in the case area and established the mathematical model according to the quantity of abandoned water as well as the actual and the theoretical output curve of the power station to quantify the abandoned water into energy loss. We then classifi...

Peer-to-peer (P2P) energy trading is being considered as an effective approach to manage the distributed energy resources (DER) and provide secure and economical demand-supply balance in the face of variable electric loads and intermittent renewable generations through the local matching of energy demand and supply. However, the existing P2P energy trading mechanisms either neglect the flexibility value of energy storage units (ES) in mitigating demand-supply imbalances or fail to adequately incentivize prosumers to participate in P2P energy trading actively. By using cooperative game theory, this work manages P2P energy trading through constructing an energy grand coalition and an easily implementable core Tâtonnement algorithm is developed to suitably reward prosumers for incentivizing their sustainable participation in the grand coalition. The convergence and stability of the proposed algorithm are theoretically proven while case studies numerically demonstrate its convergence, effectiveness and local demand-supply balancing and RES absorption ability.

Bi-level optimization constitutes the most common mathematical methodology for modeling the decision-making process of strategic generation companies in deregulated electricity markets. However, previous models neglect the physical non-convex operating characteristics of generation units, due to their inherent inability to capture binary decision variables in their representation of the market clearing process, rendering them problematic in the context of markets with complex biding and unit commitment clearing mechanisms. Aiming at addressing this fundamental limitation, this paper delves into deep reinforcement learning-based approaches by proposing a novel deep Q network (DQN) method and enabling explicit incorporation of these non-convexities into the bi-level optimization model. Case studies demonstrate the superior performance of the proposed method over the conventional Q-learning method in devising more profitable offering strategies.