Search

The financial market, as a complex nonlinear dynamic system frequently influenced by various factors, such as international investment capital, is very challenging to build trading strategies from the obtained market information. Deep Reinforcement Learning (DRL) combines the perceptual capability of deep learning and the control decision making capability of reinforcement learning to learn the mapping between financial market states and trading decisions by interacting with the environment. In this paper, an enhanced stock trading strategy, denominated efficient deep State-Action-Reward-State-Action (SARSA), is presented to tackle the algorithmic trading problem of determining optimal trading positions in the daily trading activities of the stock market. This algorithm is recognized for its properties of stable learning and convergence, attributes of critical significance within the financial domain, where stability assumes paramount importance, and excessive risk-taking should be averted. Furthermore, a novel deep network architecture called Bidirectional Long Short-Term Memory (BiLSTM)-Attention is introduced to address the challenge of accurately presenting the complex and volatile stock market. The BiLSTM-Attention greatly enhances the network's capacity to recognize key features and patterns in stock market data, allowing agents to focus on the most relevant aspects of the data. Evaluations on DJI, SP500, GE, IXIC datasets from January 1, 2008 to December 31, 2022 show that our efficient deep SARSA algorithm outperforms a wide range of traditional strategies (B&H, S&H, MR, TF) and DRL-based strategies (TDQN, DQN-Vanilla). For example, on the IXIC dataset, the efficient deep SARSA strategy achieves an attractive Cumulative Return (CR) of 582.17% and a Sharpe Ratio (ShR) of 1.86, outperforming all other methods. These experimental results prove the performance of our method in enhancing stock trading strategies. • Proposed the efficient deep SARSA model for algorithmic trading. • Proposed a network BiLSTM-Attention to extract key features in stock data. • The proposed efficient deep SARSA outperforms other baseline methods (TDQN etc.). [ABSTRACT FROM AUTHOR]

An increasing number of studies have shown the effectiveness of using deep reinforcement learning to learn profitable trading strategies from financial market data. However, a single-agent model is not sufficient to handle complex financial scenarios. To address this problem, a novel approach called Multi-Agent Double Deep Q-Network (Later called MADDQN) is proposed in this study, which reasonably balances the pursuit of maximum revenue and the avoidance of risk under the multi-agent reinforcement learning framework by innovatively employing two different agents represented respectively by two time-series feature extraction networks, TimesNet, and the Multi-Scale Convolutional Neural Network. Furthermore, to achieve a more generalized model suitable for different underlying assets, a mixed dataset containing three major U.S. stock indexes is collected. And the proposed model has been pre-trained in this dataset and subsequently refined for the specified asset. The results from experiments on five different stock indices show that the proposed MADDQN has an average cumulative return of 23.08%, outperforming the other baseline methods. Besides, the multi-agent model demonstrates its advantage in balancing the risk and revenue, in comparison with the single-agent models. Additionally, The generalization experiments confirm that the proposed MADDQN method after pre-training in the proposed mixed dataset could be stably transferred to the other underlying assets with a refinement. These findings indicate that the proposed framework not only achieves good performance in complex financial market environments but also is able to generalize robustly across different scenarios in various markets. • Proposed a deep reinforcement learning model based on TimesNet. • Proposed a framework for integrating agent strategies with different preferences. • The proposed method outperforms single-agent and baseline methods. • Pre-trained a multi-agent reinforcement learning model on a mixed dataset. [ABSTRACT FROM AUTHOR]

Research on algorithmic trading using reinforcement learning has become increasingly popular in recent years. Although most of the current reinforcement learning methods are employed to train the agent for some kind of modeling or data problem, it is worthwhile to explore in aligning agents with human behavior in applications as crucial as financial trading. Achieving such consistency by incorporating human expert experience into agent behavior is a key for potential improvements in this field. Imitation learning learns directly from examples of humans or other agents performing tasks. However, using imitation learning alone suffers from the problem of transitionally fitting expert example data. By combining the advantages of imitation learning and the Advantage Actor–Critic method, the Human Alignment Advantage Actor–Critic (HA3C) algorithm is proposed, to enhance single-asset trading strategy. First, by adding daily and weekly frequency trading data as input features to TimesNet, which is specifically designed to extract correlated temporal patterns from time-series data, it can capture both short-term and long-term features, thus capturing time-series features more comprehensively. Second, an expert action labeling method is proposed to train a strategy prediction network through supervised learning of behavior imitation. Third, a pre-trained strategy network is transferred to balance the exploration and exploitation of the agent's behavior. Imitation learning techniques leverage finance-specific knowledge to enhance algorithmic trading consistency. This approach enables algorithms to mimic and adapt human decision-making patterns in finance, ultimately improving overall performance. This paper introduces a novel return-based function that efficiently balances short-term and long-term returns over flexible time horizons. It considers the maximum return from different positions and uses flexible time windows to capture trends while maximizing returns. Finally, evaluation on six commonly used datasets, such as DJI and SP500, demonstrates the advantages of the proposed HA3C algorithm compared with other classical and reinforcement learning-based strategies. Notably, on the HSI dataset, the HA3C strategy significantly outperforms other methods, achieving an impressive cumulative return of 681.55% and a Sharpe ratio of 5.07. These results show the superior performance of the HA3C algorithm in enhancing stock trading strategies and its potential to impact algorithmic trading consistency through aligning agent behavior with human expertise. • The proposed HA3C method integrates reinforcement and imitation learning. • TimesNet network combines daily and weekly data and excels in expert policy. • A novel reward function maximizes returns with short/long-term balance. • HA3C converges faster and performs better with pre-trained expert policy network. • HA3C significantly outperforms the classical/DRL strategies on six datasets. [ABSTRACT FROM AUTHOR]