Proceedings of the Federated Conference on DOI: 10.

15439/2018F170
Computer Science and Information Systems pp. 839–844 ISSN 2300-5963 ACSIS, Vol. 15

Performance evaluation of trading strategies in multi-agent systems

– Case of A-Trader
Jerzy Korczak*, Marcin Hernes**
*
International University of Logistics and Transport, Wrocław, Poland
**
Wrocław University of Economics, Poland
e-mail:{jerzy.korczak, marcin.hernes}ue.wroc.pl

Abstract—The article presents the problem related to usability is to be selected. The idea is to "set the bar" at such
evaluation of Forex trading strategies in multi-agent systems. a level that it would not be too low, because then the result
The ratios based on financial measures cannot be assumed to would be unsatisfactory, and not too high, because it might
be only evaluation criteria because other aspects determining
effectiveness of the strategies, such as, for instance, investment be unattainable for a trader. Note that in this case, the trader
risk, statistics on winning, and lost transactions, transaction can set the "bar" higher and higher in sequence, which will
costs, should also be taken into consideration. The aim of this bring him closer to the optimal value. Therefore, the ratios
paper is to review the general financial investments based measures cannot be assumed as the only evaluation
performance measures in relation to the performance analysis criterion because other aspects having influence on the
of trading strategies. The characteristics of the commonly used
effectiveness of the strategies, such as, for instance,
performance measures are outlined. The discussion will be
illustrated by solutions developed in the trading support investment risk [12], statistics on winning and lost
system, called A-Trader system. The performance analysis in transactions as well as transaction costs should also be taken
A-Trader is detailed on real FOREX quotations. into consideration.
The aim of this paper is to review the general financial
I. INTRODUCTION investments performance measures in relation to the

T ODAY the multi-agents systems are increasingly being

used as trading support on Forex market [1, 2]. Agents
operating in such systems provide strategies for open/close
performance analysis of trading strategies. The discussion
will be illustrated by solutions developed and provided by
the A-Trader system [13]. A-Trader system is composed of
long/short positions with the use of various investments the agents capable of generating independent trading
methods and techniques. There arises the need of constant decisions on FOREX market. It allows for High Frequency
evaluation of the agent performance to guarantee satisfactory Trading (HFT). It is realized in near real time (the notion of
benefits to the trader. Many of the performance evaluation "near real time" can be understood as a very short delay
methods [e.g. 2, 3, 4] and practical solutions (e.g. between the last quote and the time of generated trading
MetaTrader [5], Plus500 [6], MetaStock [7] or NinjaTrader decision; usually 10-250 ms) and characterized by high
[8]) are based on ratios of return (e.g. rate of return, gross speed, short-term positions, it concentrates attention on price
profit, and the number of unprofitable transactions). formation process using sophisticated algorithms based on
Investment decisions are made under conditions of risk efficient and robust indicators and modern IT [14]. High
and uncertainty. In the case of a trading decision, it is frequency traders take decisions on the basis of real-time
difficult to talk about the optimal decision as the decision quotes changes in order to achieve satisfactory rate of return.
bringing the highest possible rate of return for the investor, In the first part of this paper, the characteristics of
rather should be said about the decision bringing a performance measures are outlined. Next, the methods of
satisfactory rate of return for the trader under a given level performance analysis in the trading support system, called
of other factors (e.g. risk). Many authors [e.g. 9, 10] have A-Trader, are detailed. In the final part of the article,
drawn attention to the fact that making an optimal decision conclusions and future works are presented.
is in practice very difficult in a situation of risk and
uncertainty. If, on the other hand, we adopt the principle of II. PERFORMANCE MEASURES FOR FINANCIAL INVESTMENT
satisfactory benefits, referred to in the literature as the There are many performance measures in related works.
principle of subjective expected utility [11], decision-making These measures was developed in economics, management,
process becomes less complicated. According to this and finance, both by researchers and practitioners. The
principle, for each alternative, its expected usefulness can be related works [15, 16] divide performance measures for
determined, and then the alternative that has the most financial investments into three main groups:

c
IEEE Catalog Number: CFP1885N-ART 2018, PTI 839
840 PROCEEDINGS OF THE FEDCSIS. POZNAŃ, 2018

 measures based on ratios of excess returns (ratios- E(f) – arithmetic average of the risk-free rate of
based), return,
 measures based on systematic risk measured by factor O(r) – standard deviation of rates of return.
models (risk-based),
 measures based on endogenous benchmarks derived
from portfolio theory (benchmarks-based).  Treynor Ratio
The major differences between these groups, but also 𝐸(𝑟)−𝐸(𝑓)
𝑇= (2)
𝛽(𝑟)
between specific measures refer to the definition of risk. The
next part of the section presents characteristics of particular where:
groups. E(r) – arithmetic average of the rate of return,
E(f) – arithmetic average of the risk-free rate of
A. Measures based on ratios of excess returns return,
Ratio-based performance measures specify the return per  (r) – beta coefficient of rates of return.
unit of risk. Ratio-based performance measures are usually
easy to compute and have only low data requirements. These
measures are of high relevance in practical applications and  The Kappa ratio
are frequently used in publications [12, 15, 17,18]. 𝐸(𝑟)−𝐸(𝑓)
𝑇= 𝑛 (3)
All ratios of return-based measures follow a similar √𝐿𝑃𝑀(𝑟)
schema: a measure of the return of asset in excess of the where:
return on the benchmark is divided by a measure of the E(r) – arithmetic average of the rate of return,
investment risk of asset. The most popular are the E(f) – arithmetic average of the risk-free rate of
following:: return,
 arithmetic rate of return, LPM(r) – lower partial moments of rates of return.
 logarithmic rate of return,
 the number of transaction,
 Omega ratio
 gross profit, 𝑛
√𝐻𝑃𝑀(𝑟)
 gross loss, Ω= 𝑛 (4)
√𝐿𝑃𝑀(𝑟)
 total profit,
where:
 the number of profitable transactions, HPM(r) – higher partial moments of rates of return.
 the number of profitable transactions in a row, LPM(r) – lower partial moments of rates of return.
 the number of unprofitable transactions in a row.
All of them are available in A-Trader.
 Average coefficient of variation
B. Measures based on systematic risk measured by factor
models 𝑠
𝑉= ∗ 100%
Risk-based performance measures adjust for risk by |𝐸(𝑟)|
computing the spread between actual returns and a .
hypothetical benchmark return which is determined [17]. (5)
These measures indicate whether the trader was able to where:
beat the benchmark, strictly speaking, they do not allow for V – average coefficient of variation,
comparison of different investment products because risk- s – average deviation of the rates of return,
based performance measures are subject to manipulation by E(r) – arithmetic average of the rates of return.
leverage. For identification of relevant and meaningful risk
factors and computation of “fair” or expected returns, risk-  Jensen Model
based performance measures draw heavily from the asset 𝐽𝑀 = 𝐸(𝑟) − (𝐸(𝑓) +  (𝑟) ∙ (𝐸(𝑚) − 𝐸(𝑓))). (6)
pricing literature. This group contains measures, such as [17, where:
19]: E(r) – arithmetic average of the rate of return,
E(f) – arithmetic average of the risk-free rate of
 Sharpe Ratio return,
E(m) – arithmetic average of the realized return of the
𝐸(𝑟)−𝐸(𝑓) appropriate market index,
𝑆= ∙ 100% (1)
|𝑂(𝑟)| E(f) – arithmetic average of the risk-free rate of
where: return,
E(r) – arithmetic average of the rate of return,  (r) – beta coefficient of rates of return.
MARCIN HERNES, JERZY KORCZAK: PERFORMANCE EVALUATION OF TRADING STRATEGIES IN MULTI-AGENT SYSTEMS 841

 Value at Risk 𝑇𝑀 = ∑𝑚
𝑗=1 𝐶𝑜𝑣 (𝑤𝑗𝑡, 𝑟𝑗𝑡 ) (10)
The measure known as a value exposed to the risk
- that is the maximum possible loss of the market where:
value that a financial instrument can bear in a specific 𝑤𝑗𝑡 – weight of asset j at time t,
timeframe and at a given confidence level. 𝑟𝑗𝑡 – corresponding excess return of asset j at time t,

𝑉𝐴𝑅 = 𝑃 ∗ 𝑂 ∗ 𝑘 (7) Performance measures presented in this section allow for

where: a wide range of evaluation of investment strategies on Forex.
P – the initial capital, There are many other measures in the related works,
O – volatility - standard deviation of rates of return however we have tried to select such as are more often used
during the period , in practice and which can be implemented in High-
k – the inverse of the standard normal cumulative Frequency trading systems due to low computational
distribution (assumed confidence level 95%, the complexity.
value of k is 1,65). In A-Trader, a characteristic based models are available
Some of these measures are available in A-Trader (Sharpe (based on buy and hold and random walk benchmarks).
ratio, average coefficient of variation and Value et Risk). Next part of paper presents method for performance
evaluation in A-Trader multi-agent system.
C. Measures based on endogenous benchmarks derived
from portfolio information
This group of measures usually compares the return of
III. PERFORMANCE EVALUATION METHOD IN A-TRADER
each security in the portfolio to the return of a “benchmark”
security in order to determine abnormal performance. The In general, A-Trader system is composed of the agents
comparable securities are selected based on characteristics, capable of generating independent trading decisions on
or they are derived from portfolio in another time period. FOREX market. It should be noted that decisions can be
Consequently, data requirements are higher for these consistent or contradictory, e.g. two independent agents may
models, while the statistical concepts are relatively simple generate buy and sell decision at the same time [20, 21]. The
[15, 16]. trading opportunities are provided by consensual advice,
generated by multiple software agents that use technical and
 Characteristic-Based Models fundamental analysis as well as behavioral sentiments [22].
Characteristic-Based Models are interpreted as Trading agents in the A-Trader form the investment
portfolio-weighted sum of the differences in returns strategies, which advise recommended open and closed
between the stocks and the benchmark portfolios, and positions for online FOREX traders. There are many
can be calculated, for example by following equation: strategies implemented as Supervisor Agents, such as:
𝐶𝑀𝑡 = ∑𝑚 𝑏  Basic Strategy,
𝑗=1 𝑤𝑗𝑡 (𝑟𝑗𝑡 − 𝑟𝑗𝑡−1 ) (8)
 Consensus,
where:  Candle genetic algorithm,
𝑤𝑗𝑡 – weight of asset j at time t,  Kohonen network,
𝑟𝑗𝑡 – corresponding excess return of asset j,  Growing neural gas,
𝑏  Fundamental back propagation network,
𝑟𝑗𝑡−1 – the return on a benchmark portfolio that is
 Evolutionary algorithm.
matched to asset j measured in t − 1.
 Deep Learning
 Holdings-Based Models Supervisor Agent is the most important agent in A-Trader.
These models define managerial skill as a co- Its goal is to generate profitable trading advice, on the basis
variation between portfolio weights and returns of single of three groups of The Supervisor Agent coordinates
stocks taking into consideration an omega ratio. functioning of the other agents (which form a given strategy)
presented, and to provide the final advice to the trader. Its
𝐻𝑀 = ∑𝑚 𝑗=1 𝐶𝑜𝑣 (𝑤𝑗𝑡, 𝑟𝑗𝑡 |Ω𝑡 ) (9)
other task include resolving conflicts between agents [23].
The strategies are permanently evaluated by Supervisor
where:
Agents, and those with the highest evaluation value can be
𝑤𝑗𝑡 – weight of asset j at time t,
taken by default or chosen by the trader.
𝑟𝑗𝑡 – corresponding excess return of asset j at time t,
The performance analysis in A-Trader is carried out with
Ω𝑡 – omega ratio at time t. the consideration of the following measures (ratios):
 rate of return (ratio x1),
 Trade-Based Models
 number of transactions,
These models define managerial skill as a co-
 gross profit (ratio x2),
variation between portfolio weights and returns of single
stocks.  gross loss (ratio x3),
842 PROCEEDINGS OF THE FEDCSIS. POZNAŃ, 2018

 total profit (ratio x4), and the agent's efficiency is directly proportional to the
 number of profitable transactions (ratio x5), function value.
 number of profitable consecutive transactions Figure 1 presents performance evaluation panel in A-
(ratio x6), Trader. The upper part of window presents information
related to open/close positions generated by strategy in a
 number of unprofitable consecutive transactions
given period. The profitable ones are marked on green and
(ratio x7),
unprofitable ones are marked on red. The bottom part of
 Sharpe ratio (ratio x8) window presents performance evaluation values related to
 average coefficient of variation (ratio x9) selected positions (it is possible to mark all positions, or only
 Value at Risk (ratio x10) selected positions).
 the average rate of return per transaction (ratio Referring to the evaluation analysis related to particular
x11). measures performed in other systems (mentioned in section
There are many ways of defining the performance 1), as previously underlined, these systems only offer the
evaluation function. For the purpose of comparison of the functions calculating the rates of return based ratios. It
agents' performance, the following simple evaluation should be noted that evaluation, in most cases, is performed
function has been proposed: "manually" by the trader. This work has many
𝑦 = (𝑎1 𝑥1 + 𝑎2 𝑥2 + 𝑎3(1− 𝑥3 ) + 𝑎4 𝑥4 + 𝑎5 𝑥5 + 𝑎6 𝑥6 + inconveniences. Due to its time consumption, the trader can
𝑎7 (1 − 𝑥7 ) + 𝑎8 𝑥8 + 𝑎9 (1 − 𝑥9 ) + 𝑎10 (1 − 𝑥10 ) + 𝑎11 𝑥11 use only selected measures of performance, and choice of
(11) these measures may be narrow. Also the trader acting under
where xi denote the normalized values of particular time pressure may select inadequate measures, and, in
performance measures from x1 to x11. It was adopted in the consequence, important financial losses may be generated.
test that coefficients a1 to a11=1/11. In addition, it is very difficult to have valid current
It should be mentioned that these coefficients may be knowledge on online trading (the trader’s knowledge, in
modified with the use of, for instance, an evolution-based very turbulent market conditions, may be outdated or/and
method, or they could be determined by the trader in incomplete). These issues imply that systems operating in
accordance with their preferences (for instance the trader real time are very limited.
may determine whether they are interested in higher rate of As has been mentioned, the evaluation function used in A-
return with accompanying higher risk level or lower risk Trader enables the evaluation of performance of specific
level but accepting a lower rate of return). strategies. These operations are made automatically, in time
The output of the function is a value in the range [0..1], close to real time, by the Supervisor Agent which may then

Fig. 1. Performance evaluation panel in A-Trader.

Source: Own work.
MARCIN HERNES, JERZY KORCZAK: PERFORMANCE EVALUATION OF TRADING STRATEGIES IN MULTI-AGENT SYSTEMS 843

suggest to the investor taking final decisions on the basis of REFERENCES

strategy with the highest level of performance. In addition, [1] M. Aloud, E.P.K. Tsang and R. Olsen, “Modelling the FX Market
enabling the user to change coefficients ai and ratios xi Traders' Behaviour: An Agent-based Approach”, [in] Alexandrova-
parameters of the evaluation function allows for considering Kabadjova B., S. Martinez-Jaramillo, A. L. Garcia-Almanza & E.
Tsang (eds.), Simulation in Computational Finance and Economics:
their preference concerning the criterion of importance of Tools and Emerging Applications, IGI Global, 2012, pp. 202-228.
particular evaluation ratios. The performance evaluation DOI: 10.4018/978-1-4666-2011-7.ch015.
function also considers the transaction costs with the [2] R.P. Barbosa and O. Belo, “Multi-Agent Forex Trading System”,
assumption that this reflects the relationship between the [in] Agent and Multi-agent Technology for Internet and Enterprise
Systems, Studies in Computational Intelligence Volume 289, 2010,
number of transactions and the average rate of return from pp. 91-118. https://doi.org/10.1007/978-3-642-13526-2_5.
the transaction. However, this simple principle cannot be [3] L. Mendes, P. Godinho and J. Dias, “Forex trading system based on a
adopted because a large number of transactions has impact genetic algorithm”, J. J Heuristics 18(627), 2012,
on the reduction of the strategy's efficiency level, especially https://doi.org/10.1007/s10732-012-9201-y
[4] A. Shmilovici, Y. Kahiri, I. Ben-Gal, et al., “Measuring the
for the transactions with a high rate of return. Efficiency of the Intraday Forex Market with a Universal Data
A-Trader uses only selected measures because using Compression Algorithm”, Computational Economics 33(131), 2009,
largest number of ratios require higher computing resources https://doi.org/10.1007/s10614-008-9153-3
than are now available (in HFT this measures must be [5] MetaTrader5, https://www.metatrader5.com
[6] Plus500, https://www.plus500.com/
calculated near real time). In future, we plan to use the cloud [7] MetaStock, https://www.metastock.com/
computing resources, then we will be in a position to [8] NinjaTrader, https://www.ninjatraderbrokerage.com/get_started/forex
implement a larger number of measures. It should be noted [9] F. Gul and W. Pesendorfer, “Hurwicz expected utility and subjective
that the number of performance measures in A-Trader is not sources”, Journal of Economic Theory, V 159, Part A
[10] R. Pettigrew, “Risk, rationality and expected utility theory”,
limited and they can be added to evaluation function in an Canadian Journal of Philosophy, 45:5-6, 798-826, DOI:
easy way. 10.1080/00455091.2015.111961, 2016.
[11] J. Shanteau and A. Pingenot, “Subjective expected utility theory”,
[in] M. W. Kattan (Ed.), Encyclopedia of medical decision making
(pp. 1084-1086). Thousand Oaks, CA: SAGE Publications Ltd.,
IV. CONCLUSION 2009, doi: 10.4135/9781412971980.n312.
The strategies in the A-Trader system open/close [12] K. Jajuga and T. Jajuga, “Inwestycje: Instrumenty finansowe, ryzyko
finansowe, inżynieria finansowa”, PWN, Warszawa 2000.
independent long/short position use multiple criteria of [13] J. Korczak, M. Hernes and M. Bac, “Risk avoiding strategy in
trading performance, which belong to three groups of multi-agent trading system”, [in] Proceedings of the 2013 Federated
performance measures: ratios-based, risk based, and Conference on Computer Science and Information Systems / Ganzha
Maria, Maciaszek Leszek, Paprzycki Marcin ( red. ), IEEE, 2013.
benchmark-based. As a consequence, this enables the trader DOI: 978-1-4673-4471-5/$25.00.
to compose an evaluation function according to their [14] J. Korczak, M. Hernes and M. Bac, “Collective Intelligence
preferences and to apply to the strategies of the best Supporting Trading Decisions on FOREX Market”, [in] Nguyen N.,
Supervisor Agents. The results presented in our previous Papadopoulos G., Jędrzejowicz P., Trawiński B., Vossen G. (eds)
Computational Collective Intelligence. ICCCI 2017. Lecture Notes in
research [13, 14, 21, 22, 24, 25] allow us to come to the Computer Science, vol 10448. Springer, Cham, 2017.
conclusions that there is no universally accepted evaluation https://doi.org/10.1007/978-3-319-67074-4_12.
function nor universal measures. The choice of measures and [15] P. Lückoff, “Mutual Fund Performance and Performance Persistence:
The Impact of Fund Flows and Manager Changes”, Wiesbaden:
the composition of the evaluation function is highly Gabler Verlag 2011.
dependent on trader preferences and trading market. We [16] Y. Yao and R. Zhang, “Empirical Research on Efficiency Measure of
have already demonstrated that the level of performance of Financial Investment in Education Based on SE-DEA”. [in] Cao BY.,
Liu ZL., Zhong YB., Mi HH. (eds) Fuzzy Systems & Operations
particular strategies changes depending on prevailing Research and Management. Advances in Intelligent Systems and
FOREX market situation. Based on the obtained results, Computing, vol 367. Springer, Cham, 2016.
there is no one strategy which definitely dominates over the https://doi.org/10.1007/978-3-319-19105-8_35
[17] O.K. Hussain, T.S. Dillon, F.K. and Hussain, E.J. Chang, “Risk
others. Assessment Phase: Financial Risk Assessment in Business
The use of this performance evaluation function allows Activities”, [in] Risk Assessment and Management in the Networked
for automatic setting of the best strategy in time close to real Economy. Studies in Computational Intelligence, vol 412. Springer,
Berlin, Heidelberg, 2013. https://doi.org/10.1007/978-3-642-28690-
time, which has, in turn, a positive influence on investment 2_6
effectiveness. [18] Z. Qiu, “Discussion of Investment Analysis Method in the New
Future works should concern, among others, Round of the China Stock Bull Market”, [in] Li M., Zhang Q., Zhang
implementation of other performance evaluation measures J., Li Y. (eds) Proceedings of 2015 2nd International Conference on
Industrial Economics System and Industrial Security Engineering.
(presented in section 2), development of an evolution Springer, Singapore, 2016. https://doi.org/10.1007/978-981-287-655-
method for determining ai coefficients into the A-Trader 3_40.
system, and implementation of cognitive agents performing [19] D. Hu, , G. Schwabe and X. Li, “Systemic risk management and
investment analysis with financial network analytics: research
analysis experts' opinions in the scope of forecasts referring opportunities and challenges”, Financial Innovation (2015) 1: 2.
to quotations on the FOREX market.. https://doi.org/10.1186/s40854-015-0001-x.
[20] J. Korczak, M. Bac, K. Drelczuk and A. Fafuła, “A-Trader -
Consulting Agent Platform for Stock Ex-change Gamblers”, [in]
Proc. FedCSIS, Wrocław, pp.963-968, 2012.. DOI: 978-83-60810-
51-4/$25.00.
844 PROCEEDINGS OF THE FEDCSIS. POZNAŃ, 2018

[21] J. Korczak and M. Hernes, “Deep Learning for Financial Time Series [23] J. Korczak, M. Hernes and M. Bac, “Fuzzy Logic as Agents’
Forecasting in A-Trader System”, [in] Proceedings of the 2017 Knowledge Representation in A-Trader System”, [in] Information
Federated Conference on Computer Science and Information Technology for Management. Federated Conference on Computer
Systems / Ganzha Maria, Maciaszek Leszek, Paprzycki Marcin Science and Information Systems, ISM 2015 and AITM 2015, Lodz,
(eds.), Annals of Computer Science and Information Systems, vol. Poland, Revised Selected Papers / Ziemba Ewa ( red. ), Lecture
11, Polskie Towarzystwo Informatyczne, Institute of Electrical and Notes in Business Information Processing, vol. 243, 2016, Springer,
Electronics Engineers, 2017. DOI: 10.15439/2017F449. ISBN 978-3-319-30527-1, ss. 109-124, 2015.
[22] M. Hernes and J. Sobieska-Karpińska , “Application of the [24] J. Korczak, M. Hernes and M. Bac, “Risk avoiding strategy in multi-
consensus method in a multi-agent financial decision support agent trading system”, [in] Proceedings of the 2013 Federated
system”, Information Systems and e-Business Management 14 (1), Conference on Computer Science and Information Systems / Ganzha
Springer Berlin Heidelberg, 2016, DOI: 10.1007/s10257-015-0280-9. Maria, Maciaszek Leszek, Paprzycki Marcin ( red. ), IEEE, 2013.