Neural Network Forecasting of Time Series

DOI: 10.14529/ctcr190412 NEURAL NETWORK FORECASTING OF TIME SERIES A.O. Golovenko, [email protected], A.A. Kopyrkin, [email protected] South Ural State University, Chelyabinsk, Russian Federation In the work, we built a predictive neural network to successfully predict several main classes of radar data, as well as economic indicators. It is a two-layer neural network feedforward network based on the backpropagation error algorithm. The results of forecasting real radio signals. Based on the results of the forecast, it turned out that the neural network ensures the accuracy of the short-term forecast. In this article, we describe the procedures for selecting characteristics for learning a neural network, justifying the choice of the structure of the neural network, training and the results obtained. Time series forecasting is currently an important topic, as it has a wide range of applications (radar, medicine, socio-economic sphere, energy, risk management, engineering applications, etc.). Analysis of works in the field of long-term forecasting of non-deterministic signals showed that at the moment the least studied is the neural network long-term forecasting. The use of neural networks for long-term forecasting is based on their ability to approximate nonlinear functions, the accumulation of history and its application in forecasting and learning ability. The work was based on the method of neural network forecasting using a two-layer network with direct distribution. The implemented neural network can be used to predict real signals of different frequency bands. This study can be very useful in medicine, geodesy, Economics and other areas. Keywords: RF signals, neural network, time series forecasting, MATLAB. Introduction By neural networks are meant computational structures that simulate simple biological processes that are somehow related to a person's brain activity. They are parallel systems capable of adaptive learning by analyzing input effects. The elementary converter in these networks is the neuron. Artificial NN are built according to the principles of organization and functioning of their biological analogs. They are able to solve a wide range of problems of image recognition, identification, prediction, optimization, management of complex objects. Special attention should be paid to the use of neural network technologies (dynamic neural networks are the most relevant now) to improve the processing of radar information in difficult conditions, which require high computing power, when the dynamics of changing external conditions are very high and traditional approaches to the creation of processing systems can’t provide the required result [1]. In addition to the ability to solve a new class of problems, neural networks have a number of significant advantages. First, it is resistance to input noise, which allows the use of neural networks in highprecision communication systems. Such an opportunity for neural networks appears due to the so-called training. After training, they are able to ignore the inputs to which noise data is fed. Neural networks are able to function correctly, even if the input is noisy. Secondly, adaptation to change. This means that with small changes in the environment, the neural network is able to adapt to changes. Consider a neural network that predicts the growth / fall in prices on the exchange. However, gradually, day by day, the situation on the market is changing. If the network did not adapt to these changes, it would stop giving the right answers in a week. But artificial neural networks, learning from the data, each time adjust to the environment. Third, it is fault tolerance. They can give out correct results even at considerable damage of components making them. Fourth, ultra-high speed. The computer executes commands in sequence. However, in the head of a person, each neuron is a small processor (which receives a signal, converts it, and sends it to the output). And there are billions of such processors in our heads. We get a giant network of distributed computations. The signal is processed by neurons simultaneously. This property potentially manifests itself in artificial neural networks. If you have a multi-core computer, this property will be executed. For single-core computers, there will be no noticeable difference [2]. 124 Bulletin of the South Ural State University. Ser. Computer Technologies, Automatic Control, Radio Electronics. 2019, vol. 19, no. 4, pp. 124–131 Головенко А.О., Копыркин А.А. Нейросетевое прогнозирование временных рядов However, neural networks have a serious drawback. It is worth noting that neural networks, despite a wide range of tasks that they can solve, still remain only useful additional functionality. In the first place there are always computer programs. The remarkable news is that by integrating conventional software algorithms and neural networks you can almost completely get rid of all potential flaws. Studies on artificial neural networks are related to the fact that the way information is processed by the human brain is fundamentally different from the methods used by conventional digital computers. The brain is an extremely complex, nonlinear, parallel computer. He has the ability to organize his structural components, called neurons, so that they can perform specific tasks (such as pattern recognition, sensory processing, motor functions) many times faster than most high-speed modern computers can. Proceeding from all described above it becomes clear why at this stage of development of methods and methods of programming so much attention is paid to neural network programming [3]. The prediction of time series is currently a topic, as it has a wide field of application (radiolocation, medicine, socio-economic sphere, energy, risk management, engineering applications, etc.). Analysis of works in the field of long-term forecasting (22, 24, 25) of nondeterministic signals showed that at the moment the least studied is neural network long-range forecasting. The use of neural networks for long-term forecasting is based on their ability to approximate non-linear functions, to accumulate history and its application in forecasting and learning ability [4]. Model description The structure of an artificial neuron can be represented in the form of a model containing three basic elements: 1) A set of synapses characterized by their weights. In particular, the signal xj at the input of the synapse j, connected with the neuron k, is multiplied by the weight wkj; 2) Totalizer, which adds input signals, weighted relative to the corresponding neuron synapses; 3) The activation function limits the amplitude of the output signal of the neuron [5]. In mathematical terms, the functioning of a neuron is expressed by the following equations (1): =∑ , = ( + ). (1) The graphically nonlinear model of a neuron is show in Fig. 1. The Hecht-Nielsen theorem proves the representability of a function of several variables of a fairly general type by means of a two-layer neural network with direct complete connections to the N components of the input signal, 2N + 1 components of the first (“hidden” layer) with previously known limited activation functions (for example, sigmoidal) and M components of the second layer with unknown activation functions. The theorem, therefore, in a non-constructive form proves the solvability of the problem of representing a function of a sufficiently arbitrary form on HC and indicates for each problem the minimum values of the number of network neurons needed for the solution. We show multilayer neural network in Fig 2. Fig. 1. Nonlinear model of a neuron In the study of the neural network, the following formula for estimating the necessary number of synaptic weights Lw in a multilayer (two-layer) network with a sigmoidal activation function was used [6]. ≤ ≤ + 1 ( + + 1) + . (2) Having estimated the necessary number of weights, we can calculate the number of neurons in the hidden layers. Since a two-layer network was investigated, we will give the formula (3) as applied to it [7]. = . (3) Having defined a certain network structure, it is necessary to find the optimal values of the weight coefficients. This stage is called teaching the NA. The task of learning is to find (synthesize) some optimal function. It requires long calculations and represents an iterative procedure, the iteration number of which can reach from 10 to 100 000 000 [8]. There are the following basic methods of learning the neural network: local optimization, stochastic optimization, global optimization, training with the teacher, algorithm for back propagation of the error. We chose the algorithm of global optimization, since it is Вестник ЮУрГУ. Серия «Компьютерные технологии, управление, радиоэлектроника». 2019. Т. 19, № 4. С. 124–131 125 Краткие сообщения the speedy and less resource-consuming [9]. But this algorithm has its own drawback, it does not allow you to train large dimensions. The last component of the neural network is the activation function, which, as noted earlier, normalizes the output values and determines the output signal of the neuron as a function of the induced local field [10, 11]. We chose the hyperbolic tangent function (a kind of sigmoidal activation function), which is generally described by expression [12] ( ) = α tanh ( ) , ( , ) > 0. (4) Algorithm of the neural network: 1) I/O coding: neural networks can only work with numbers; 2) Data normalization: the results of the neuro analysis should not depend on the choice of units of measurement; 3) Preprocessing data: removing obvious regularities from the data makes it easier for neural networks to identify nontrivial regularities; 4) Neural network training; 5) Adaptation; 6) Evaluation of the significance of the prediction. For the study, a two-layer neural network of direct propagation Fig. 2. Multilayer neural network with a sigmoidal activation function and an algorithm for learning with sequential connections the Error Back Propagation was used [13] (Fig. 3). The number of neural network epochs equal to 175 (the number of views of the training sample) was determined, which is optimal for predicting the strip non-deterministic process. Increasing or decreasing this value can result in retraining or under-training, resulting in chaotic effects in the network (the network will be hypersensitive or vice versa) [14]. The dependence of the exact forecast length on the length of the training sample was studied for a nondeterministic process passed through a bandpass filter with fo = 530 Hz and Δf = 60 Hz (Fig. 4). We introduce such a condition, the sample is considered accurate if the prediction error is 10% or less. Fig. 3. The model of a two-layer network of direct propagation In the MATLAB programming environment, a neural network algorithm was implemented [15], a nondeterministic process with a duration longer than the training sequence of the neural network was set up to enable analysis of the accuracy of the forecast. For the training sequence of the neural network, equal to To = 3000 samples, we get that the length of the accurately predicted sequence is 17 samples (Fig. 5). When the length of the training sequence is halved (To = 1500 samples), the number of predicted samples is exactly (under given conditions) equal to 14 (Fig. 6). With the length of the training sequence equal to 750 samples, we obtained an accurately predicted sequence (with errors included) equal to 10 samples. (Fig. 7). With a training sequence length of 350 samples, an exactly predicted sequence of 6 samples was obtained (Fig. 8). 126 Bulletin of the South Ural State University. Ser. Computer Technologies, Automatic Control, Radio Electronics. 2019, vol. 19, no. 4, pp. 124–131 Головенко А.О., Копыркин А.А. Нейросетевое прогнозирование временных рядов Magnitude Response (dB) 0 -10 Magnitude (dB) -20 -30 -40 -50 -60 -70 -80 -90 0 100 200 300 400 Frequency (Hz) 500 600 700 Fig. 4. Filter frequency response Fig. 5. Band random process and its forecast at To = 3000 Fig. 6. Band random process and its forecast at To = 1500 Fig. 7. Band random process and its forecast at To = 750 Fig. 8. Band random process and its forecast at To = 350 Further reduction of the training sequence does not lead to positive results, the mean square error for the predicted sequence of the first counts exceed 10%, and the predicted sequence become chaotic (Fig. 9). Table 1 show the dependence of the forecast length on the length of the training sample. The dependence of the length of the forecast on the length of the training sample No. 1 2 3 4 5 To 3000 2000 1500 750 350 Тn 17 15 14 10 6 Table 1 To/Тn 176,5 133,3 107,1 75,0 58,3 Вестник ЮУрГУ. Серия «Компьютерные технологии, управление, радиоэлектроника». 2019. Т. 19, № 4. С. 124–131 127 Краткие сообщения Fig. 9. Dependence of Tn on To Since the nondeterministic process itself is poorly predicted, it is suggested to use the wavelet transform to increase the length of the exact signal forecast. As a result of this transformation, the original signal is decomposed into several narrowband signals, which are predicted and, by addition, are converted to the original signal. We showed this results on Tables 2 and 3. Table 2 Dependence of the change in the forecast length on the bandwidth of the signal at To = 3000 ∆f 1 10 20 60 80 100 120 160 300 400 600 800 Тn 960 96 50 16 13 10 11 11 6 5 4 3 ∆f·Тn 960 960 1000 960 1040 1000 1320 1760 1800 2000 2400 2400 Table 3 Dependence of the change in the forecast length on the bandwidth of the signal at To = 1500 ∆f 1 10 20 60 80 100 120 160 300 400 600 800 Тn 900 90 45 15 11 9 8 7 5 4 3 2 ∆f·Тn 900 900 900 900 880 900 960 1120 1500 1600 1800 1600 In Fig. 10 and Fig. 11 shows the Δf ·Tn dependence of the bandwidth of a nondeterministic signal. It is seen from the diagram that the dependence has a linear character on three segments. It should also be noted that with increasing spectral processing of the signal up to the Δf ~ 3 kHz band, the neural network gives an accurate prediction of only one value in advance. From this we can conclude that it becomes a neural network with a short-term forecast. Fig. 10. Dependence Δf ·Tn on the signal bandwidth at To = 3000 Fig. 11. Dependence Δf ·Tn on the signal bandwidth at To = 1500 128 Bulletin of the South Ural State University. Ser. Computer Technologies, Automatic Control, Radio Electronics. 2019, vol. 19, no. 4, pp. 124–131 Головенко А.О., Копыркин А.А. Нейросетевое прогнозирование временных рядов The dependence of the accuracy of the forecast on the central frequency was studied as follows: from the random sequence, the value of the sine at the central frequency was subtracted, and a prediction of this sequence was provided. Further the forecast developed with the same sinusoid. Thus, the accuracy of the forecast at long-range frequencies was improved. In the study of this dependence, the following regularity was observed: if the passband of the filter is left unchanged and the center frequency of this band is shifted to the right by Δf / 2, then the accuracy and length of the forecast will remain constant, although the probability of occurrence of “emissions” in the predicted sequence will appear. The above analysis allows us to choose a rational approach to solving the problem of forecasting nondeterministic processes that are limited in the band, as well as other time series. The choice of this or that method allows to improve the quality of the forecast and its duration. The research was based on the method of neural network prediction using a two-layer network with direct propagation. Conclusions Due to the use of realistic radio signals of the neural technology market implemented in a correctly designed, optimized and trained neural network, it was possible to provide a sufficiently high (95–97%) accuracy of the short-term forecast of the listed data. These results allow us to say that neural network technologies are a powerful tool for solving forecasting problems and allow making forecasts with high accuracy. Further development of this work is planned in the field of financial, seismology (determining the likelihood of an earthquake) and medicine (determining the probability of occurrence of cardiac diseases by ECG). References 1. Haykin S. Neyronnye seti: polnyy kurs [Neural Networks: Full Course]. Moscow, Williams Publishing House, 2006. 1104 p. 2. Borovikov V.P. Neyronnye seti Statistica neural networks: Metodologiya i tekhnologii sovremennogo analiza dannyh [Neural Networks Statistica Neural Networks: Methodology and Technologies of Modern Data Analysis]. Moscow, “Hotline –Telecom” Publ., 2008. 392 p. 3. Osovsky S. Neyronnye seti dlya obrabotki informatsii [Neural Networks for Information Processing]. Moscow, Finance and Statistics Publ., 2002. 344 p. 4. Rudakov A.S. [Approaches to Solving the Task of Forecasting Time Series Using Neural Networks]. Business Informatics, 2008, vol. 6, no. 4, pp. 29–34. (in Russ.) 5. Kashirina I.L. [About the Methods of Forming Neural Network Assemblies in Problems of Forecasting Financial Time Series]. Bulletin of VSU, 2009, no. 2, pp. 116–119. (in Russ.) 6. Rudoy G.I. [The Choice of the Activation Function in the Prediction of Neural Networks]. Machine Learning and Data Analysis, 2011, no. 1, pp. 16–39. (in Russ.) 7. Antipov O.I., Neganov V.A. Analiz i prognozirovanie povedeniya vremennykh ryadov: bifurkatsii, katastrofy, prognozirovanie i neyronnye seti [Analysis and Forecasting of Time Series Behavior: Bifurcations, Catastrophes, Forecasting and Neural Networks]. Moscow, Radiotekhnika Publ., 2011. 350 p. 8. Tatuzov A.L. Neyronnye seti v zadachakh radiolokatsii [Neural Networks in Radar Tasks]. Moscow, Radiotekhnika Publ., 2009. 432 p. 9. Dudnik P.I., Cheresov Yu.I. Aviacionnye radiolokacionnye ustrojstva [Aviation Radar Devices]. MAA named by N.E. Zhukovskiy Publ., 1986. 533 p. 10. Nicholls John, Martin Robert, Wallas Bruce, Fuchs Paul Ot neyrona k mozgu [From Neuron to the Brain]. Moscow, Editorial URSS Publ., 2003. 672 p. 11. Zenin A.V. [Analysis of Pattern Recognition Methods]. Young Scientist, 2017, no. 16, pp. 125– 130. (in Russ.) 12. Darovskikh S.N., Zvonov B.M., Safin. D.K. [Model of Compression of Sound Information on Neural Networks]. Reports of USSR Science Academy. Series Biology, 1990, no. 9, pp. 99–104. (in Russ.) 13. Antipov O.I., Neganov V.A. [Fractal Analysis of Nonlinear Systems and Construction of Predictive Neural Networks on its Basis]. Physics of Wave Processes and Radio Engineering Systems, 2010, vol. 13, no. 3, pp. 54–63. (in Russ.) Вестник ЮУрГУ. Серия «Компьютерные технологии, управление, радиоэлектроника». 2019. Т. 19, № 4. С. 124–131 129 Краткие сообщения 14. Golovenko A.O. [Neural Network Time Series Forecasting]. AETERNA, 2015, no. 2, pp. 59–63. (in Russ.) 15. Golovenko A.O., Ragozin A.N. [Forecasting Economic Time Series Using Neural Network Technologies]. Scientific and Analytical Economic Journal, 2016, no. 1. (in Russ.) Received 8 July 2019 __________________________________________________________________ УДК 004.032.26 DOI: 10.14529/ctcr190412 НЕЙРОСЕТЕВОЕ ПРОГНОЗИРОВАНИЕ ВРЕМЕННЫХ РЯДОВ А.О. Головенко, А.А. Копыркин Южно-Уральский государственный университет, г. Челябинск, Россия Построена прогностическая нейронная сеть для успешного прогнозирования нескольких основных классов радиолокационных данных, а также экономических показателей. Это двухслойная нейронная сеть прямой связи, основанная на алгоритме ошибки обратного распространения. Приведены результаты прогнозирования реальных радиосигналов. По результатам прогноза оказалось, что нейронная сеть обеспечивает точность краткосрочного прогноза. В данной статье описываются процедуры выбора характеристик для обучения нейронной сети, обосновывается выбор структуры нейронной сети, обучение и полученные результаты. Прогнозирование временных рядов в настоящее время является важной темой, так как имеет широкий спектр применения (радиолокация, медицина, социально-экономическая сфера, энергетика, управление рисками, инженерные приложения и другие сферы применения). Анализ работ в области долгосрочного прогнозирования недетерминированных сигналов показал, что на данный момент наименее изученной является нейросеть долгосрочного прогнозирования. Использование нейронных сетей для долгосрочного прогнозирования основано на их способности аппроксимировать нелинейные функции, накоплении истории и ее применении в прогнозировании и обучаемости. Ключевые слова: радиочастотные сигналы, нейронная сеть, прогнозирование временных рядов, MATLAB. Литература 1. Хайкин, Саймон. Нейронные сети: полный курс: пер. с англ. / Саймон Хайкин. – 2-е изд. – М.: Вильямс, 2006. – 1104 с. 2. Боровиков, В.П. Нейронные сети Statistica neural networks: Методология и технологии современного анализа данных / В.П. Боровиков. – М.: Горячая линия – Телеком, 2008. – 392 с. 3. Осовский, С. Нейронные сети для обработки информации / С. Осовский. – М.: Финансы и статистика, 2002. – 344 с. 4. Рудаков, А.С. Подходы к решению задачи прогнозирования временных рядов с помощью нейронных сетей / А.С. Рудаков // Бизнес-информатика. – 2008. – Т. 6, № 4. – С. 29–34. 5. Каширина, И.Л. О методах формирования нейросетевых ассамблей в задачах прогнозирования финансовых временных рядов / И.Л. Каширина // Вестник ВГУ. – 2009. – № 2. – С. 116–119. 6. Рудой, Г.И. Выбор функции активации при прогнозировании нейронными сетями / Г.И. Рудой // Машинное обучение и анализ данных. – 2011. – № 1. – С. 16–39. 7. Антипов, О.И. Анализ и прогнозирование поведения временных рядов: бифуркации, катастрофы, прогнозирование и нейронные сети / О.И. Антипов, В.А. Неганов. – М.: Радиотехника, 2011. – 350 с. 8. Татузов, А.Л. Нейронные сети в задачах радиолокации / А.Л. Татузов. – М.: Радиотехника, 2009. – 432 с. 130 Bulletin of the South Ural State University. Ser. Computer Technologies, Automatic Control, Radio Electronics. 2019, vol. 19, no. 4, pp. 124–131 Головенко А.О., Копыркин А.А. Нейросетевое прогнозирование временных рядов 9. Дудник, П.И. Авиационные радиолокационные устройства / П.И. Дудник, Ю.И. Чересов. – Изд-во «ВВИА им. Н.Е. Жуковского», 1986. – 533 с. 10. От нейрона к мозгу: пер. с англ. / Джон Николлс, Роберт Мартин, Брюс Валлас, Пол Фукс. – М.: Едиториал УРСС, 2003. – 672 с. 11. Зенин, А.В. Анализ методов распознавания образов // Молодой ученый. – 2017. – № 16. – С. 125–130. 12. Модель сжатия звуковой информации в нейронных сетях / С.Н. Даровских, Б.М. Звонов, Д.К. Сафини др. // Изв. АН СССР. Сер. Биология. – 1990. – № 9. – С. 99–104. 13. Антипов, О.И. Фрактальный анализ нелинейных систем и построение на его основе прогнозирующих нейронных сетей / О.И. Антипов, В.А. Неганов // Физика волновых процессов и радиотехнические системы. – 2010. – Т. 13, № 3 – С. 54–63. 14. Головенко, А.О. Нейросетевое прогнозирование временных рядов / А.О. Головенко // АЭТЕРНА. – 2015. – № 2. – С. 59–63. 15. Головенко, А.О. Прогнозирование экономических временных рядов с использованием нейросетевых технологий / А.О. Головенко, А.Н. Рагозин // Научно-аналитический экономический журнал. – 2016. – № 1. Головенко Антон Олегович, аспирант, лаборант кафедры «Системное программирование», Южно-Уральский государственный университет, г. Челябинск; [email protected]. Копыркин Александр Андреевич, аспирант кафедры «Инфокоммуникационные технологии», Южно-Уральский государственный университет, г. Челябинск; [email protected]. Поступила в редакцию 8 июля 2019 г. ОБРАЗЕЦ ЦИТИРОВАНИЯ FOR CITATION Golovenko, A.O. Neural Network Forecasting of Time Series / A.O. Golovenko, A.A. Kopyrkin // Вестник ЮУрГУ. Серия «Компьютерные технологии, управление, радиоэлектроника». – 2019. – Т. 19, № 4. – С. 124–131. DOI: 10.14529/ctcr190412 Golovenko A.O., Kopyrkin A.A. Neural Network Forecasting of Time Series. Bulletin of the South Ural State University. Ser. Computer Technologies, Automatic Control, Radio Electronics, 2019, vol. 19, no. 4, pp. 124–131. DOI: 10.14529/ctcr190412 Вестник ЮУрГУ. Серия «Компьютерные технологии, управление, радиоэлектроника». 2019. Т. 19, № 4. С. 124–131 131