Deep Reinforcement Learning in Portfolio

Management
Zhipeng Liang ∗† ,Kangkang Jiang∗† ,Hao Chen ∗† ,Junhao Zhu ∗† ,Yanran Li ∗† ,
∗ Likelihood Technology
† Sun Yat-sen University

{liangzhp6, jiangkk3, chenhao348, zhujh25, liyr8}@mail2.sysu.edu.cn

arXiv:1808.09940v1 [q-fin.PM] 29 Aug 2018

Abstract—In this paper, we implement two state-of-art continu- based search algorithm [2]. Saud Almahdi et al. extended
ous reinforcement learning algorithms, Deep Deterministic Policy recurrent reinforcement learning and built an optimal vari-
Gradient (DDPG) and Proximal Policy Optimization (PPO) in able weight portfolio allocation under the expected maximum
portfolio management. Both of them are widely-used in game
playing and robot control. What’s more, PPO has appealing drawdown [3]. Xiu Gao et al. used absolute profit and relative
theoretical propeties which is hopefully potential in portfolio risk-adjusted profit as performance function to train the system
management. We present the performances of them under differ- respectively and employ a committee of two network, which
ent settings, including different learning rate, objective function, was found to generate appreciable profits from trading in the
markets, feature combinations, in order to provide insights for foreign exchange markets [4].
parameter tuning, features selection and data preparation.
Index Terms—Reinforcement Learning; Portfolio Manage- Thanks to the development in deep learning, well known
ment; Deep Learning; DDPG; PPO for its ability to detect complex features in speech recogni-
tion, image identification, the combination of reinforcement
I. I NTRODUCTION learning and deep learning, so called deep reinforcement
learning, has achieved great performance in robot control,
Utilizing deep reinforcement learning in portfolio manage-
game playing with few efforts in feature engineering and
ment is gaining popularity in the area of algorithmic trading.
can be implemented end to end [5]. Function approximation
However, deep learning is notorious for its sensitivity to neural
has long been an approach in solving large-scale dynamic
network structure, feature engineering and so on. Therefore,
programming problem [6]. Deep Q Learning, using neural
in our experiments, we explored influences of different op-
network as an approximator of Q value function and replay
timizers and network structures on trading agents utilizing
buffer for learning, gains remarkable performance in playing
two kinds of deep reinforcement learning algorithms, deep
different games without changing network structure and hyper
deterministic policy gradient (DDPG) and proximal policy op-
parameters [7]. Deep Deterministic Policy Gradient(DDPG),
timization (PPO). Our experiments were conveyed on datasets
one of the algorithms we choose for experiments, uses actor-
of China and America stock market. Our codes can be viewed
critic framework to stabilize the training process and achieve
on github1 .
higher sampling efficiency [8]. Another algorithm, Proximal
II. S UMMARY Policy Optimization(PPO), turns to derive monotone improve-
ment of the policy [9].
This paper is mainly composed of three parts. First, port-
Due to the complicated, nonlinear patterns and low signal
folio management, concerns about optimal assets allocation
noise ratio in financial market data, deep reinforcement learn-
in different time for high return as well as low risk. Sev-
ing is believed potential in it. Zhengyao Jiang et al. proposed
eral major categories of portfolio management approaches
a framework for deep reinforcement learning in portfolio man-
including ”Follow-the-Winner”, ”Follow-the-Loser”, ”Pattern-
agement and demonstrated that it can outperform conventional
Matching” and ”Meta-Learning Algorithms” have been pro-
portfolio strategies [10]. Yifeng Guo el at. refined log-optimal
posed. Deep reinforcement learning is in fact the combination
strategy and combined it with reinforcement learning [11].
of ”Pattern-Matching” and ”Meta-Learning” [1].
However, most of previous works use stock data in America,
Reinforcement learning is a way to learn by interacting
which cannot provide us with implementation in more volatile
with environment and gradually improve its performance by
China stock market. What’s more, few works investigated the
trial-and-error, which has been proposed as a candidate for
influence of the scale of portfolio or combinations of different
portfolio management strategy. Xin Du et al. conducted Q-
features. To have a closer look into the true performance and
Learning and policy gradient in reinforcement learning and
uncover pitfalls of reinforcement learning in portfolio man-
found direct reinforcement algorithm (policy search) enables
agement, we choose mainstream algorithms, DDPG and PPO
a simpler problem representation than that in value function
and do intensive experiments using different hyper parameters,
1 https://github.com/qq303067814/Reinforcement-learning-in-portfolio- optimizers and so on.
management- The paper is organized as follows: in the second section
 P
we will formally model portfolio management problem. We i wi,t−1 = 1. We assume initial wealth is P0 . Definitions of
will show the existence of transaction cost will make the state, action and reward in portfolio management are as below.
problem from a pure prediction problem whose global op- • State(s): one state includes previous open, closing, high,
timized policy can be obtained by greedy algorithm into a low price, volume or some other financial indexes in a
computing-expensive dynamic programming problem. Most fixed window.
reinforcement learning algorithms focus on game playing and • Action(a): the desired allocating weights, at−1 =
robot control, while we will show that some key characters (a0,t−1 , a1,t−1 , . . . , am,t−1 )T is the allocating vector at
in portfolio management requires some modifications. The Pn
period t − 1, subject to the constraint i=0 ai,t−1 = 1.
third part we will go to our experimental setup, in which Due to the price movement in a day, the weights vector
we will introduce our data processing, our algorithms and at−1 at the beginning of the day would evolve into wt−1
our investigation into effects of different hyper parameters at the end of the day:
to the accumulated portfolio value. The fourth part we will
yt−1  at−1
demonstrate our experiment results. In the fifth part we would wt−1 =
come to our conclusion and future work in deep reinforcement yt−1 · at−1
learning in portfolio management.

III. P ROBLEM D EFINITION

Given a period, e.g. one year, a stock trader invests into a
set of assets and is allowed to reallocate in order to maximize
his profit. In our experiments, we assume that the market is
continuous, in other words, closing price equals open price
Fig. 1. The evolution of weights vector
the next day. Each day the trading agent observes the stock
market by analyzing data and then reallocates his portfolio. In • Reward(r): the naive fluctuation of wealth minus trans-
addition, we assume that the agent conducts reallocation at the action cost. The fluctuation of wealth is aTt−1 · yt−1 .
end of trade days, which indicates that all the reallocations can In the meanwhile, transaction
Pmcost should be subtracted
be finished at the closing price. In addition, transaction cost, from that, which equals µ i=1 |ai,t−1 − wi,t−1 |. The
which is measured as a fraction of transaction amount, has equation above suggests that only transactions in stocks
been taken into considerations in our experiments. occur transaction cost. Specifically, we set µ = 0.25%.
Formally, the portfolio consists of m+1 assets, including In conclusion, the immediate reward at time t-1 as:
m risky assets and one risk-free asset. Without depreciation, m
X
we choose money as the risk-free asset. The closing price rt (st−1 , at−1 ) = log(at−1 ·yt−1 −µ |ai,t−1 −wi,t−1 |)
of ith asset after period t is vi,t close
. The closing price of i=1
all assets comprise the price vector for period t as vtclose . .
Modeling as a Markovian decision process, which indicates The introduction of transaction cost is a nightmare to some
the next state only depends on current state and action. Tuple traditional trading strategy, such as follow the winner, follow
(S, A, P, r, ρ0 , γ) describes the entire portfolio management the loser etc. Even can we predict precisely all stock price
problem where S is a set of states, A is a set of actions, in the future, deriving the optimal strategy when the period
P : S × A × S → R is the transition probability distribution, is long or the scale of portfolio is large, is still intractable.
r : S → R is the reward function. ρ0 : S → R is the Without transaction cost, greedy algorithm can achieve optimal
distribution of the initial state s0 and γ ∈ (0, 1) is the discount profits. To be specific, allocating all the wealth into the asset
factor. which has the highest expected increase rate is the optimal
It’s worth to note that in Markovian decision process, most policy in such a naive setting. However, the existence of
objective
PTfunctions take the form of discount rate, which is transaction cost might turn action changing too much from
t
R = t=1 γ r(st , at ). However, in the area of portfolio previous weight vector into suboptimal action if the transaction
management, due to the property that the wealth accumulated cost overweights the immediate return.
by time t would be reallocated QT in time t+1, indicating that the Although rich literatures have discussed Markovian decision
wealth at time T, PT = t=1 P0 rt is continued product form process, portfolio management is still challenging due to
but not summation. A sightly modification would be needed, its properties. First and foremost, abundant noise included
which is to take logarithm of the return to transform continued in the stock data leads to distorted prices. Observations of
product form into summation. stock prices and financial indexes can hardly reflect the states
To clarify each item in the Markovian decision pro- underneath. Providing inefficient state representations for the
cess, we make some notations here. Define yt = algorithm would lead to disastrous failure in its performance.
vt v1,t vm,t T
vt−1 = (1, v1,t−1 , . . . , vm,t−1 ) as the price fluctuating vec- What’s more, the transition probability of different states is
tor. wt−1 = (w0,t−1 , w1,t−1 , . . . , wm,t−1 )T represents the still unknown. We must learn environment before we attempt
reallocated weight at the end of time t − 1 with constraint to solve such a complex dynamic programing problem.
 Although buying and selling stocks must be conducted by The Bellman Equation allows us to compute it by recursion:
hands, here we still adapt continuous assumption. In fact
Qπ (st , at ) = Ert ,st+1 ∼E [r(st , at )+γEat+1 ∼π [Qπ (st+1 , at+1 )]]
when wealth is much more than the prices of stocks, such
a simplification would not lose much generation. For a deterministic policy which is a function µ : S → A,
IV. D EEP R EINFORCEMENT LEARNING the above equation can be written as:
Reinforcement learning, especially combining with state-of- Qπ (st , at ) = Ert ,st+1 ∼E [r(st , at ) + γ[Qµ (st+1 , µ(st+1 ))]]
art deep learning method is therefore thought to be a good
To be specific, Q-learning adapts greedy policy which is:
candidate for solving portfolio problem. Reinforcement learn-
ing is a learning method, by which the agent interacts with µ(s) = arg max Q(s, a)
a
the environment with less prior information and learning from
the environment by trail-and-error while refining its strategy at Deep reinforcement learning uses neural network as the
the same time. Its low requirements for modeling and feature Q-function approximator and some methods including replay
engineering is suitable for dealing with complex financial buffer are proposed to improve the convergence to the optimal
markets. What’s more, deep learning has witnessed its rapid policy. Instead of using iterations to derive the conventional
progress in speech recognition and image identification. Its Q-value function, the function approximator, parameterized by
outperformance with conventional methods has proven its θQ , is derived by minimizing the loss function below:
capability to capture complex, non-linear patterns. In fact, L(θQ ) = Est ∼ρβ ,at ∼β,rt ∼ [(Q(st , at |θQ ) − yt )2 ]
different methods using neural network in designing trading
algorithms have been proposed. where
Compared with solely using deep learning or reinforcement yt = r(st , at ) + γQ(st+1 , µ(st+1 )|θQ )
learning in portfolio management, deep reinforcement learning
It’s worth to note here that yt is calculated by a separate
mainly has three strengths.
target network which is softly updated by online network.
First, with market’s information as its input and allocating
This simple change moves the relatively unstable problem
vector as its output, deep reinforcement learning is an totally
of learning the action-value function closer to the case of
artificial intelligent methods in trading, which avoids the hand-
supervised learning, a problem for which robust solutions
made strategy from prediction of the future stock price and can
exist. This is another method to improve convergence.
fully self-improved.
When dealing with continuous action space, naively imple-
Second, deep reinforcement learning does not explicitly
menting Q-learning is intractable when the action space is
involve predictions towards stock performance, which has been
a large due to the ”curse of dimensionality”. What’s more,
proven very hard. Therefore, less challenges would hinder the
determining the global optimal policy in an arbitrary Q-
improvement in reinforcement learning performance.
value function may be infeasible without some good features
Third, compared with conventional reinforcement learning,
guaranteed such as convex.
deep reinforcement learning approximates strategy or value
The answer of DDPG to address the continuous control
function by using neural network, which can not only include
problem is to adapt policy gradient, in which DDPG consists
the flexibility of designing specific neural network structure
of an actor which would directly output continuous action.
but also prevent so called ”curse of dimensionality”, enabling
Policy would then be evaluated and improved according to
large-scale portfolio management.
critic, which in fact is a Q-value function approximator to
Several continuous reinforcement learning methods have
represent objective function. Recall the goal of Markovian
been proposed, such as policy gradient, dual DQN, Deep De-
decision process: derive the optimal policy which maximize
terministic Policy Gradient and Proximal Policy Optimization.
the objective function. Parameterized by θ, we can formally
We conduct the latter two algorithms in our experiments to test
write it as:
their potential in portfolio management.
A. Deep Deterministic Policy Gradient τ = (s1 , a1 , s2 , a2 , . . . )
X
Deep Deterministic Policy Gradient(DDPG) is a combi- J(πθ ) = Eτ ∼pθ (τ ) [ γ t r(st , at )]
nation of Q-learning and policy gradient and succeed in t

using neural network as its function approximator based on πθ∗ = arg max J(πθ )
πθ
Deterministic Policy Gradient Algorithms.[12] To illustrate X
its idea, we would briefly introduce Q-learning and policy = arg max Eτ ∼pθ (τ ) [ γ t r(st , at )]
πθ
t
gradient and then we would come to DDPG.
Q-learning is a reinforcement learning based on Q-value = arg max Eτ ∼pθ (τ ) [r(τ )]
πθ
function. To be specific, a Q-value function gives expected
Z
accumulated reward when executing action a in state s and = arg max πθ (τ )r(τ )dτ
πθ
follow policy π in the future, which is:
In deep reinforcement learning, gradient descent is the most
Qπ (st , at ) = Eri ≥t,si >t E,ai >t π [Rt |st , at ] common method to optimize given objective function, which
is usually non-convex and high-dimensional. Taking derivative Algorithm 1 DDPG
of the objective function equals to take derivative of policy. 1: Randomly initialize actor µ(s|θ µ ) and critic Q(s, a|θ Q )
0 0
Assume the time horizon is finite, we can write the strategy 2: Create Q0 and µ0 by θ Q → θ Q ,θ µ → θ µ
in product form: 3: Initialize replay buffer R
4: for i = 1 to M do
πθ (τ ) = πθ (s1 , a1 , . . . , sT , aT )
5: Initialize a UO process N
T
Y 6: Receive initial observation state s1
= p(s1 ) πθ (at |st )p(st+1 |st , at )
7: for t = 1 to T do
t=1
8: Select action at = µ(st |θµ ) + Nt
However, such form is difficult to make derivative in terms 9: Execute action at and observe rt and st+1
of θ. To make it more computing-tractable, a transformation 10: Save transition (st ,at ,rt ,st+1 ) in R
has been proposed to turn it into summation form: 11: Sample a random minibatch of N transitions
∇θ πθ (τ ) (si ,ai ,ri ,si+1 ) in R
0 0
∇θ πθ (τ ) = πθ (τ ) 12: Set yi = ri + γQ0 (si+1 , µ0 (si+1 |θµ )|θQ ) P
πθ (τ )
13: Update critic by minimizing the loss:L = N1 i (yi −
= πθ (τ )∇θ log πθ (τ )
Q(si , ai |θQ ))2
T
X 14: Update actor policy by policy gradient:
∇θ log πθ (τ ) = ∇θ (log p(s1 ) + log πθ (at |st ) + log p(st+1 ))
t=1 ∇θ µ J
T 1 X
≈ ∇θµ Q(s, a|θQ )|s=st ,a=µ(st |θµ ) ∇θµ µ(s|θµ )|st
X
= ∇θ log πθ (at , st ) N i
t=1
(1)
Therefore, we can rewrite differentiation of the objective
function into that of logarithm of policy: 15: Update the target networks:
0 0
∇J(πθ ) = Eτ ∼πθ (τ ) [r(τ )] θQ → τ θQ + (1 − τ )θQ
= Eτ ∼πθ (τ ) [∇θ log πθ (τ )r(τ )] 0 0
θµ → τ θµ + (1 − τ )θµ
T
X T
X
= Eτ ∼πθ (τ ) [( ∇θ log πθ (at |st ))( γ t r(st , at ))] 16: end for
t=1 t=1 17: end for
In deep deterministic policy gradient, four networks are
required: online actor, online critic, target actor and target
critic. Combining Q-learning and policy gradient, actor is Optimization(TRPO) [13], we would introduce TRPO first and
the function µ and critic is the Q-value function. Agent then PPO.
observe a state and actor would provide an ”optimal” action in TRPO finds an lower bound for policy improvement so that
continuous action space. Then the online critic would evaluate policy optimization can deal with surrogate objective function.
the actor’s proposed action and update online actor. What’s This could guarantee monotone improvement in policies.
more, target actor and target critic are used to update online Formally, let π denote a stochastic policy π : S×A → [0, 1],
critic. which indicates that the policy would derive a distribution in
Formally, the update scheme of DDPG is as below: continuous action space in the given state to represent all the
For online actor: action’s fitness. Let
∞
X
η(π) = Es0 ,a0 ,... [ γ t r(st )]
∇θµ J ≈ Est ∼ρβ [∇θµ Q(s, a|θQ )|s=st ,a=µ(st |θµ ) ] t=0
= Est ∼ρβ [∇a Q(s, a|θQ )|s=st ,a=µ(st ) ∇θµ µ(s|θµ )|s=st ] s0 ∼ ρ0 (s0 ), at ∼ π(at |st ),st+1 ∼ P (st+1 , at+1 |st , at )
For online critic, the update rule is similar. The target actor Following standard definitions of the state-action value
and target critic are updated softly from online actor and online function Qπ , the value function Vπ and the advantage function
critic. We would leave the details in the presentation of the as below:
algorithm: X∞
Vπ (st ) = Eat ,st+1 ,... [ γ l r(st+l )]
B. Proximal Policy Optimization l=0
Most algorithms for policy optimization can be classified ∞
X
into three broad categories:(1) policy iteration methods. (2) Qπ (st , at ) = Est+1 ,at+1 ,... [ γ l r(st+l )]
policy gradient methods and (3) derivative-free optimization l=0
methods. Proximal Policy Optimization(PPO) falls into the
second category. Since PPO is based on Trust Region Policy Aπ (s, a) = Qπ (s, a) − Vπ (s)
 The expected return of another policy π̃ over π can be ex-
pressed in terms of the advantage accumulated over timesteps:
η(πi+1 ) ≥ Mi (πi+1 )
X∞
η(π̃) = η(π) + Es0 ,a0 ,···∼π̃ [ γ t Aπ (st , at )] Therefore, we could give out the lower bound of the policy
t=0 improvement:
The above equation can be rewritten in terms of states:
η(πi+1 ) − η(πi ) ≥ Mi (πi+1 ) − Mi (πi )
∞ X
X X
η(π̃) = η(π) + P (st = s|π̃) π̃(a|s)γ t Aπ (s, a) Thus, by maximizing Mi at each iteration, we guarantee that
s
t=0 a the true objective η is non-decreasing. Consider parameterized
∞
XX X policies πθi , the policy optimization can be turned into:
= η(π) + γ t P (st = s|π̃) π̃(a|s)Aπ (s, a)
s t=0 a
X X max
= η(π) + ρπ̃ (s) π̃(a|s)Aπ (s, a) max[Lπθi−1 (πθi ) − CDKL (πθi−1 , πθi )]
πθi
s a

where ρπ̃ = P (s0 = s) + γP (s1 = s) + γ 2 P (s2 = s) + · · · However, the penalty coefficient C from the theoretical
denotes the discounted visitation frequencies of state s given result would provide policy update with too small step sizes.
policy π̃. While in the final TRPO algorithm, an alternative optimization
However, the complexity due to the reliance to policy problem is proposed after carefully considerations of the
π̃ makes the equation difficult to compute. Instead, TRPO structure of the objective function:
proposes the following local approximation.
X X maxLπθi
Lπ (π̃) = η(π) + ρπ (s) π̃(a|s)Aπ (s, a) πθi
ρπθ
s a s.t. DKLi−1 (πθi−1 , πθi ) ≤ δ
The lower bound of policy improvement, as one of the key
ρ
results of TRPO, provides theoretical guarantee for monotonic where DKL (πθ1 , πθ2 ) = Es∼ρ [DKL (πθ1 (·|s)||πθ2 (·|s))]
policy improvement: Further approximations are proposed to make the optimiza-
tion tractable. Recalled that the origin optimization problem
4γ
η(πnew ) ≥ Lπold (πnew ) − α2 can be written as :
(1 − γ)2
X X
where max ρπθi−1 (s) πθi (a|s)Aθi−1 (s, a)
πθ
 = max |Aπ (s, a)| s a
s,a
α= DTmax
V (πold , πnew )
After some approximations including importance sampling,
= max DT V (πold (·|s)||πnew (·|s)) the final optimization comes into:
s
1 πθi (a|s)
P
DT V (p||q) = 2i |pi −qi | is the total variation divergence max Es∼ρπθi−1 ,a∼q [ Aπθi−1 (s, a)]
distance between two discrete probability distributions. π θi q(a|s)
Since DKL (p||q) ≥ DT V (p||q)2 , we can derive the fol-
lowing inequation, which is used in the construction of the s.t. Es∼ρπθi−1 [DKL (πθi−1 (·|s)||πθi (·|s))] ≤ δ
algorithm:

max So here comes the PPO[9]: it proposed new surrogate

η(π̃) ≥ Lπ (π̃) − CDKL (π, π̃)
objective to simplify TRPO. One of them is clipped surrogate
where objective which we choose in our experiments. Let us denote
4γ r(θ) = πππθ (a|s)
(a|s) . The clipped surrogate objective can be
old
C= written as:
(1 − γ)2
max
DKL (π, π̃) = max DKL (π(·|s)||π̃(·|s))
s

The proofs of above equations are available in [13] LCLIP (θ) = E[min(r(θ)A, clip(r(θ), 1 − , 1 + )A)]
To go further into the detail, let Mi (π) = Lπi (π) −
max
CDKL (πi , π). Two properties would be uncovered without This net surrogate objective function can constrain the
much difficulty as follow: update step in a much simpler manner and experiments show
it does outperform the original objective function in terms of
η(πi ) = Mi (πi ) sample complexity.
Algorithm 2 PPO In order to derive a general agent which is robust with
1: Initialize actor µ : S → Rm+1 and different stocks, we normalize the price data. To be specific,
σ : S → diag(σ1 , σ2 , · · · , σm+1 ) we divide the opening price, closing price, high price and low
2: for i = 1 to M do price by the close price at the last day of the period. For
3: Run policy πθ ∼ N (µ(s), σ(s)) for T timesteps and missing data which occurs during weekends and holidays, in
collect (st , at , rt ) order to maintain the time series consistency, we fill the empty
0
Estimate advantages Ât = t0 >t γ t −t rt0 − V (st )
P
4: price data with the close price on the previous day and we also
5: Update old policy πold ← πθ set volume 0 to indicate the market is closed at that day.
6: for j = 1 to N do
7: Update actor policy by policy gradient:
X
∇θ LCLIP
i (θ)
i
B. network structure
8: Update critic by:
T
X 2
∇L(φ) = − ∇Ât Motivated by Jiang et al., we use so called Identical In-
t=1 dependent Evaluators(IIE). IIE means that the networks flow
independently for the m+ assets while network parameters
9: end for
are shared among these streams. The network evaluates one
10: end for
stock at a time and output a scaler to represent its preference
to invest in this asset. Then m+1 scalers are normalized by
V. E XPERIMENTS softmax function and compressed into a weight vector as the
A. Data preparation next period’s action. IIE has some crucial advantages over
an integrated network, including scalability in portfolio size,
Our experiments are conducted on China Stock data and data-usage efficiency and plasticity to asset collection. The
America Stock data from investing.com, wind and Shinging- explanation can be reviewed in[10] and we are not going to
Midas Private Fund. We select two baskets of stocks with low illustrate them here.
correlation or even negative correlation from these markets
to demonstrate our agent’s capability to allocate between We find that in other works about deep learning in portfolio
different assets. In order to hold our assumption, we choose management, CNN outperforms RNN and LSTM in most
stocks with large volume so that our trades would not affect cases. Howevero, different from Jiang et al., we alternate
the market. In China stock market, we choose 18 stocks in CNN with Deep Residual Network. The depth of the neural
order to test the algorithm in large-scale portfolio management network plays an important role in its performance. However,
setting. In America stock market we choose 6 stocks. What’s conventional CNN network is stopped from going deeper
more, we choose last 3 years as our training and testing because of gradient vanishment and gradient explosion when
period, with 2015/01/01−2016/12/31 as training period and the depth of the networks increases. Deep residual network
2017/01/01 − 2018/01/01 as testing period. The stock codes solves this problem by adding a shortcut for layers to jump
we select are as follow: to the deeper layers directly, which could prevent the network
from deteriorating as the depth adds. Deep Residual Network
market code market code has gained remarkable performance in image recognition and
China 000725 USA AAPL greatly contributes to the development of deep learning.
China 000002 USA ADBE
China 600000 USA BABA
China 000862 USA SNE
China 600662 USA V
China 002066
China 600326
China 000011
China 600698
China 600679
China 600821
China 600876
China 600821
China 000151
China 000985
China 600962

TABLE I Fig. 2. Residual Block

STOCK CODES
 Fig. 3. DDPG Network Structure in our experiments

Fig. 5. Critic loss under different actor learning rates

Fig. 4. PPO Network Structure in our experiments

DDPG PPO
Algorithm
Actor Critic Actor Critic
Optimizer Adam Adam GradientDescent GradientDescent
Learning Rate 10−3 10−1 10−3 10−3
τ 10−2 10−2 10−2 10−2
Fig. 6. Critic loss under different critic learning rates
TABLE II
H YPER PARAMETERS IN OUR EXPERIMENTS
2) Risk: Due to the limitation of training data, our rein-
forcement learning agent may underestimate the risk when
C. result training in bull market, which may occur disastrous deteriora-
tion in its performance in real trading environment. Different
1) learning rate: Learning rate plays an essential role in approaches in finance can help evaluate the current portfolio
neural network training. However, it is also very subtle. A risk to alleviate the effect of biased training data. Inspired by
high learning rate will make training loss decrease fast at the Almahdi et al. in which objective function is risk-adjusted and
beginning but drop into a local minimum occasionally, or even Jacobsen et al. which shows the volatility would cluster in a
vibrate around the optimal solution but could not reach it. A period, we modify our objective function as follow:
low learning rate will make the training loss decrease very
slowly even after a large number of epochs. Only a proper T
X
learning rate can help network achieve a satisfactory result. R= γ t (r(st , at ) − βσt2 )
Therefore, we implement DDPG and test it using different t=1
learning rates. The results show that learning rates have Pt Pm+1
significant effect on critic loss even actor’s learning rate does where σt2 = L1 t0 =t−L+1 i=1 (yi,t0 − yi,t0 )2 · wi,t and
t
yi,t0 = L1 t0 =t−L+1 yi,t0 measure the volatility of the re-
P
not directly control the critic’s training. We find that when
the actor learns new patterns, critic loss would jump. This turns of asset i in the last L day. The objective function is
indicates that the critic has not sufficient generalization ability constrained by reducing the profit from investing in highly
towards new states. Only when the actor becomes stable can volatile assets which would make our portfolio exposed in
the critic loss decreases. exceeded danger.
 Fig. 7. Comparison of portfolio value with different risk penalties(β) Fig. 8. Comparison of critic loss with different features combinations

Unfortunately, the result seems not support our modifica-

tions. We also train our agent in objective function taking form
of Sharpe ratio but it also fails. In fact, reward engineering
is one of the core topics in designing reinforcement learning
algorithms. It seems that our modification makes the objective
function too complex [15].
3) Features combination: As far as we know, few works
discuss the combinations of features in reinforcement learning.
Different from end to end game playing or robot control whose
input is pixels, in portfolio management, abundant features can
be taken into considerations. Common features include the
closing price, the open price, the high price, the low price and
volume. What’s more, financial indexes for long term analysis
such as Price-to-Earning Ratio (PE), Price to book ratio (PB)
can also provide insights into market movements.
However, adding irrelevant features would add noise and Fig. 9. Comparison of reward with different features combinations
deteriorate the training. The trade off in it is the topic of feature
selection. Therefore, we conduct experiments under different
combinations of features, which are 1. only with closing
policy seems unsuitable in such an unstationary, low signal
prices, 2. with closing and high, 3. with closing and open,
noise ratio financial market although its theoretical properties
4. with closing and low prices. The results show that feature
are appealing, including monotone improvement of policy and
combinations matter in the training process. Select closing and
higher sample efficiency.
high price could help agent gain the best performance in our
experiments.
4) Training and Testing: After experiments mentioned APV(%) Sharpe Ratio(%) Maximum Drawdown
above, we derive a satisfying set of hyper parameters and DDPG 159.904 1.379 0.164
features combinations. Under such setting, we conduct training PPO 106.520 0.867 0.107
Winner 66.484 1.411 0.160
for 1000 epochs on both China stock market and USA stock Loser 57.662 1.150 0.066
market. The result shows that training could increase accumu- UCRP 144.371 0.695 0.0482
lative portfolio value (APV) while reducing the volatility of
the returns in training data. TABLE III
Then we back test our agent on USA and China data. DDPG P ERFORMANCES OF DIFFERENT STRATEGIES
agent gains the highest APV amongst all the agents. However,
its risk is also higher than other agents with the highest
maximum drawdown. The unsatisfying performance of PPO Back test on China data is frustrated in which our agent
algorithm also uncovers the considerable gap between game seems learn nothing in training. The possible reasons will be
playing or robot control and portfolio management. Random discussed in our conclusion.
Fig. 10. Comparison of portfolio value before and after learning in training Fig. 12. Backtest on USA stock market
data of China stock market

following optimization problem:

t+H−1
X 0
(at , . . . , at+H−1 ) = arg max γ t −t r(st0 , at0 )
at ,...,at+H−1
t0 =t

What’s more, due to the fact that neural network is sensitive

to the quality of data, traditional financial data noise reduction
approaches can be utilized, such as wavelet analysis [18] and
the Kalman Filter [19]. A different approach for data pre-
processing is to combine HMM with reinforcement learning,
which is to extract the states beneath the fluctuated prices and
learning directly from them [20].
Modification of the object function can also be taken into
considerations. One direction is to adapt risk-adjust return.
Another direction we come up with experiments in designing
RL agent in game playing. In game playing, the reward
function is simple, for example, in flappy bird, the agent would
Fig. 11. Comparison of portfolio value before and after learning of America receive reward 1 when passing a pillar or receive reward -1
stock market
when drop to the ground. Complex objective function would
hinder agent from achieving desirable performance. We have
conduct a naive version of accumulative portfolio value as
VI. F UTURE WORK object function, which is to take win rate instead of absolute
return but we cannot receive satisfying improvement.

Thanks to the characteristics of portfolio management, there VII. C ONCLUSION

is still many interesting topics in combination with deep This paper applies reinforcement learning algorithms with
reinforcement learning. For future research, we will try to use continuous action space to asset allocation. We compare the
other indicators to measure the risk of our asset allocation, performances of DDPG and PPO algorithms in different
and work on the combination with conventional models in markets, hyper parameters and so on. Compared with previous
finance to make advantages of previous finance research. To works of portfolio management using reinforcement learning,
be specific, we believe model-based reinforcement as a good we test our agents with risk-adjusted accumulative portfolio
candidate in portfolio management instead of model-free [16] value as objective function and different features combinations
[17].In model-based reinforcement learning, a model of the as input. The experiments show that the strategy obtained by
dynamics is used to make predictions, which is used for DDPG algorithm can outperform the conventional strategies in
action selection. Let fθ (st ; at ) denote a learned discrete-time assets allocation. It’s found that deep reinforcement learning
dynamics function, parameterized by θ, that takes the current can somehow capture patterns of market movements even
state st and action at and outputs an estimate of the next state though it is allowed to observe limited data and features and
at time t + ∆t. We can then choose actions by solving the self-improve its performance.
 However, reinforcement learning does not gain such remark- [18] Rua, Antnio, and Lus C. Nunes. ”International comovement of stock
able performance in portfolio management so far as those in market returns: A wavelet analysis.” Journal of Empirical Finance 16.4
(2009): 632-639.
game playing or robot control. We come up with a few ideas. [19] Faragher, Ramsey. ”Understanding the basis of the Kalman filter via a
First, the second-order differentiability for the parameters in simple and intuitive derivation.” IEEE Signal processing magazine 29.5
the neural network of the output strategy and the expectation (2012): 128-132.
[20] Serban, Iulian Vlad, et al. ”The Bottleneck Simulator: A Model-
in Q-value is necessary for convergence of the algorithm. Due based Deep Reinforcement Learning Approach.” arXiv preprint
to the algorithm, we could only search for optimal policy in arXiv:1807.04723 (2018).
the second-order differentiable strategy function set, instead of [21] Rao, Anil V. ”A survey of numerical methods for optimal control.”
Advances in the Astronautical Sciences 135.1 (2009): 497-528.
in the policy function set, which might also lead to the failure
of finding globally optimal strategy.
Second, the algorithm requires stationary transition. How-
ever, due to market irregularities and government intervention,
the state transitions in stock market might be time varying.
In our experiments, deep reinforcement learning is highly
sensitive so that its performance is unstable. What’s more,
the degeneration of our reinforcement learning agent, which
often tends to buy only one asset at a time, indicates more
modifications are needed for designing promising algorithms.
ACKNOWLEDGMENT
We would like to say thanks to Mingwen Liu from Shing-
ingMidas Private Fund, Zheng Xie and Xingyu Fu from Sun
Yat-sen University for their generous guidance. Without their
support, we could not overcome so many challenges during
this project.
R EFERENCES
[1] Li, Bin, and Steven CH Hoi. ”Online portfolio selection: A survey.” ACM
Computing Surveys (CSUR) 46.3 (2014): 35.
[2] Du, Xin, Jinjian Zhai, and Koupin Lv. ”Algorithm trading using q-learning
and recurrent reinforcement learning.” positions 1 (2009): 1.
[3] Almahdi, Saud, and Steve Y. Yang. ”An adaptive portfolio trading
system: A risk-return portfolio optimization using recurrent reinforcement
learning with expected maximum drawdown.” Expert Systems with
Applications 87 (2017): 267-279.
[4] An Algorithm for Trading and Portfolio Management Using Q-learning
and Sharpe Ratio Maximization
[5] Li, Yuxi. ”Deep reinforcement learning: An overview.” arXiv preprint
arXiv:1701.07274 (2017).
[6] Sutton, Richard S., et al. ”Policy gradient methods for reinforcement
learning with function approximation.” Advances in neural information
processing systems. 2000.
[7] Mnih, Volodymyr, et al. ”Human-level control through deep reinforcement
learning.” Nature 518.7540 (2015): 529.
[8] Lillicrap, Timothy P., et al. ”Continuous control with deep reinforcement
learning.” arXiv preprint arXiv:1509.02971 (2015).
[9] Schulman, John, et al. ”Proximal policy optimization algorithms.” arXiv
preprint arXiv:1707.06347 (2017).
[10] Jiang, Zhengyao, Dixing Xu, and Jinjun Liang. ”A deep reinforcement
learning framework for the financial portfolio management problem.”
arXiv preprint arXiv:1706.10059 (2017).
[11] Guo, Yifeng, et al. ”Robust Log-Optimal Strategy with Reinforcement
Learning.” arXiv preprint arXiv:1805.00205 (2018).
[12] Silver, David, et al. ”Deterministic policy gradient algorithms.” ICML.
2014.
[13] John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, Pieter
Abbel: Trust Region Policy Optimization
[14] Jacobsen, Ben, and Dennis Dannenburg. ”Volatility clustering in monthly
stock returns.” Journal of Empirical Finance 10.4 (2003): 479-503.
[15] Dewey, Daniel. ”Reinforcement learning and the reward engineering
principle.” 2014 AAAI Spring Symposium Series. 2014.
[16] Gu, Shixiang, et al. ”Continuous deep q-learning with model-based
acceleration.” International Conference on Machine Learning. 2016.
[17] Nagabandi, Anusha, et al. ”Neural network dynamics for model-based
deep reinforcement learning with model-free fine-tuning.” arXiv preprint
arXiv:1708.02596 (2017).