IJCIET_10_01_150.pdf

International Journal of Civil Engineering and Technology (IJCIET) Volume 10, Issue 01, January 2019, pp. 1646-1656, Article ID: IJCIET_10_01_150 Available online at http://www.iaeme.com/ijciet/issues.asp?JType=IJCIET&VType=10&IType=01 ISSN Print: 0976-6308 and ISSN Online: 0976-6316 © IAEME Publication Scopus Indexed METHODS OF MEASUREMENT, EVALUATION AND PREDICTION OF METASUBJECT EDUCATIONAL RESULTS Arseniy Aleksandrovich Lebedev Laboratory of Innovations, Ltd, Kazan, Tatarstan, Russian Federation ABSTRACT In this study, the main goal is to measure, evaluate predict student’s metasubject educational results. Federal state educational standards (FSES) of primary, secondary and basic education of Russian Federation distinguish three types of educational results: personal, metasubject, subject. The FSES formulates requirements for all three types of educational results, which implies availability of methods for measuring or evaluating these results. Personal educational results are left out of framework of this SRP, since only for subject and meta-subject educational results there are more or less well-established methods of assessment and measurement. Most well-developed ones are methods for measuring and evaluating subject educational results. A huge number of different measuring and measuring instruments (tests, assignments, essays, etc.) have been developed primarily to evaluate these results. Methods and tools for measuring and evaluating metasubject educational results have been developed to a much lesser degree. This research continues those recent works on experiments, in which analysis of user activity log files is used to evaluate metasubject educational results. In the course of such experiments, the task is demonstrated to a student on a computer screen, and all user interactions with the computer are recorded into log files. Keywords: Distance learning, Method for measuring, Monitoring, online courses, Structure, Information technology, Educational outcomes prediction, Federal state educational standards, Metasubject educational results. Cite this Article: Arseniy Aleksandrovich Lebedev, Methods of Measurement, Evaluation and Prediction of Metasubject Educational Results, International Journal of Civil Engineering and Technology, 10(01), 2019, pp. 1646–1656 http://www.iaeme.com/IJCIET/issues.asp?JType=IJCIET&VType=10&IType=01 http://www.iaeme.com/IJCIET/index.asp 1646 [email protected] Arseniy Aleksandrovich Lebedev 1. INTRODUCTION Section II of the FSES of elementary general education indicates that meta-subject educational results include adoption of interdisciplinary concepts and universal learning activities (ULA) by students. The same section provides a list of metasubject results. ULAs are divided into FSES into following subgroups [1]: 1. regulatory ULAs: management of its activities; control and correction; initiative and independence; 2. communicative ULA: speech activity; cooperation skills; 3. cognitive ULA: work with information; work with training models; use of symbolic means, general solution schemes; performing logical operations of comparison, analysis, generalization, classification, establishment of analogies, summarizing concept; After examining lists of metadisciplinary educational results and description of ULA given in FSES, one can come to conclusion that these concepts are similar to the notion of problemsolving skills, the importance of development of which has been actively pointed out by foreign researchers since mid-70s of the past century [2]. According to A.A. Kuznetsova, meta-subject educational results are methods of activity that are applicable both within the framework of educational process and in solving problems in real life situations mastered by students on the basis of one, several or all subjects [3]. The sphere of automating measurement and evaluation of metasubject educational results in difference to subject educational results, is extremely young [4]. One of earliest methods for diagnosing meta-subject educational results is the TAP protocol. The paper by Yu B., Voll K., “Probing student problem solving skills in mathematical induction using a scenario-based think aloud protocol” describes the use of this method to evaluate problem solving skills in the study of mathematics. Nevertheless, the method of recordation of allegations aloud is extremely cumbersome, its automation is extremely difficult and its application to large groups of students (including taking mass open online courses) is almost impossible [5]. A peculiar simplification of the TAP method is the use of various tools for questioning students. As an example of using this method, experiments using MAI or MSLQ can be given as an example. However, the method of questioning suffers from a number of shortcomings, the most serious of which, according to Veenmann, is the difficulty of verifying obtained results [6]. A number of studies show that the survey does not reflect the use of specific problem-solving skills during completion of a task [7]. Students do not apply methods that they declared before the start of completion of tasks themselves during their completion. Moreover, many students do not remember exactly how they performed a particular task. When questioning, a student must reconstruct the course of his actions from memory, and therefore he can easily forget or distort his actions [8]. The use of telecommunication technologies in conducting experiments has become possible online over the past decade. Natural development of methods for recordation allegations aloud and questioning were obtained in connection with the development of such technologies [9]. In this case, behavior of a students is evaluated by independent experts in real time according to predetermined criteria. http://www.iaeme.com/IJCIET/index.asp 1647 [email protected] Methods of Measurement, Evaluation and Prediction of Metasubject Educational Results 2. LITERATURE REVIEW An external observer who seeks to identify the presence or absence of a metasubject educational result, virtually all aspects of student's behavior are available, but in many cases they are analyzed intuitively [10]. This was reported by many teachers participating in the survey on the GlobalLab platform. Only in some cases, an observer can accurately describe on what signs in a student's behavior he was able to conclude about absence/presence of a metasubject educational result [11]. This feature is the main reason for numerous difficulties in formalization of procedures for measuring and evaluating metasubject educational results [12]. From this point of view, the SSR has two important multidirectional features [13]: • rich means of accurate recordation of almost all user actions therein; • almost complete absence of any means of monitoring the cognitive process. On one hand, these features expand possibilities of very accurate recordation of actions performed by a user and components of behavior variation, and, on the other, they impose significant restrictions on fullness of manifestations of behavior variations in SSR [14]. The manifestation of behavior variation at the SSR level is composed of a large number of uniform user actions, which are recorded in SSR as a set of its “physical” actions (scrolling, mouse movements, etc.) [15]. Moreover, not all elements of behavior variations are expressed in SSR. It can be assumed that manifestation of the SSR level is just as poorly represented as the observed behavior variation, as a behavior variation itself weakly represents a cognitive process occurring in the student's mind [16]. Thus, in SSR, deep signs of a cognitive process, characterized by presence or absence of a metasubject educational result, pass through two filters that filter out some of these signs — through the observed behavior variation (filter 1) and its manifestation in the SSR (filter 2) [17]. The fact that there are signs sufficient to distinguish metasubject educational result after the first filter is confirmed by the fact that an experienced expert, carefully observing a student, can measure and evaluate this result (using TAP, MAI, MSLQ recordation and questioning method described therein) [18]. The high correlation of diagnostic results with use of Otter SSR by the TAP Otter group diagnostics (0.96) confirms that the second filter also lets through a number of signs sufficient to detect a metasubject educational result [19]. This prediction is also supported by the accuracy of forecasting in papers in which the subject educational result is predicted solely on the basis of data on metasubject: Prediction accuracy ranges from 69 to 83%. 3. MATERIALS AND METHODS Recently, a few works on experiments, in which analysis of user activity log files is used to evaluate metasubject educational results, began to appear. In the course of such experiments, the task is demonstrated to a student on a computer screen, and all user interactions with the computer are recorded into log files. Of particular interest for this SRP is an experiment using the Otter e-learning environment conducted by a group of Dutch researchers led by Veenmann in 2014 [20]. The main goal of Veenmann's research was to test the hypothesis of sufficient sensitivity of the method of analyzing log files to detect differences in development of problem-solving skills among students of different sex and age. http://www.iaeme.com/IJCIET/index.asp 1648 [email protected] Arseniy Aleksandrovich Lebedev The learning environment used in the experiment is a Learning-by-discovery SSR. During performance of research tasks, students should reveal the effect of 5 independent variables on the growth of otter population while conducting virtual experiments. Among these variables were following ones: the habitat area (its size and fragmentation), degree of environmental pollution, people's access to the habitat, emergence of new pairs of otters and presence of a fish diet in the winter season. These independent variables may not affect the population size (for example, people's access to a habitat), they may have a general effect on it (pollution and habitat) and may interact with other variables (for example, the habitat and appearance of new pairs). In each virtual experiment, student chooses a value for all 5 variables by clicking on appropriate icons, and then the program calculates the resulting size of otter population. All results of experiments, together with values of variables are placed in the scrolling area so that a student can view them while planning the next experiment. After 15 experiments, a student can finish working with the system, but the exit from it is the result of his free choice in this case. 4. RESULTS AND DISCUSSIONS In the course of completion of the assignment, all user actions are recorded to a log file, which is then used by algorithm for calculating the degree of development of problem-solving skills. Table 1 presents metrics that are calculated based on these log files and then used by the algorithm. Table 1. Intermediate Metrics for Problem Solving Skills in the Otter Environment Metrics Number.exp Thinktime Scrolldown Scrollup Votat.pos Votat.neg Unique.exp Description The total number of virtual experiments performed by the user. The time in seconds elapsed from the moment the last experiment was completed by the first click in a new experiment. The amount on all experiments. The frequency of scrolldown to earlier experiments. Frequency of scrollup for later experiments. The number of transitions between experiments that differ in the value of only one variable. The average number of variables changed in the next experiment minus 1. The number of unique experiments out of 48 possible ones. The total number of virtual experiments (Number.exp) is considered as a metric varying in proportion to the degree of development of problem-solving skills. The more experiments a student has conducted, more complete the general experiment is considered. Evaluation and development of new solutions provokes a student to conduct new experiments. The time elapsed from completion of the last experiment to the first click in a new experiment (Thinktime) is considered by researchers as a positive metric reflecting the student’s ability to evaluate the result, reorient and plan the next experiment. The frequency of scrolling to earlier and later experiments is also a positive metric reflecting the development of problem-solving skills. Scrolling indicates at least a student's intention to take into account previous experience when planning a new step in learning. The fourth metric, Votat.pos, reflects the number of variables that changed between experiments. Changing only one variable is a positive sign indicating a student's desire to conduct an experiment in accordance with a pre-planned scheme. http://www.iaeme.com/IJCIET/index.asp 1649 [email protected] Methods of Measurement, Evaluation and Prediction of Metasubject Educational Results The average number of variables changed between experiments is a negative metric. If more than one variable is changed between experiments, a student will not be able to attribute the resulting change to a specific variable. The systematic repetition of such an error indicates a low metasubject result. Finally, the fifth metric (Unique.exp) reflecting the degree of coverage of experimental space is certainly positive. Students with advanced problem-solving skills usually monitor the experimental space and plan their next experiment accordingly. The total degree of development of problem-solving skills is calculated according to the following algorithm: • the z-score is calculated for all seven metrics; • the sign of metric Votat.neg is inverted; • for Scrolldown and Scrollup metrics, the average Scrolling z-score is calculated; • for metrics Votat.pos and Votat.neg, the average z-score for Votat is calculated; • For 5 metrics obtained, the average z-score, which is considered to be a general measure of development of problem-solving skills, is calculated. Interestingly, the correlation of results obtained by methods of analyzing log files with results of experiments using the method of recordation reflections aloud conducted earlier by the same group of researchers amounted to 0.84 and 0.96. Despite very harsh and limited conditions, the experiment of the group of Veenmann demonstrated the possibility of successful using the journal data on user behavior in SSR for evaluation of metasubject educational results. It is also important to note that in the experiment of the group of Veenmann, researchers had to select variables and a mathematical model describing the relationship between user actions in SSR and metasubject educational result intuitively. The clear advantage of using any type of neural network compared to the approach of Veenmann's group in automating assessment and forecasting of metasubject educational results is the fact that when using a neural network, the function of dependence of educational result (including non-linear) on the recorded data is generated as a result of network training rather than being developed by a researcher. Nevertheless, results of Veenmann's group convincingly show the existence of a connection between such simple characteristics of a student's behavior such as time between tasks or the number of training experiments conducted and meta-subject results which are so difficult to evaluate. The use of data that accumulates when journaling user actions in SSR to predict metasubject educational results of students requires formulation of a hypothesis explaining the fundamental relationship between simple data on user behavior in SSR and their significance and sufficiency for evaluating and predicting metasubject educational results. Such a hypothesis should answer the question: why SSR journaling data is enough to reflect at least some problem-solving skills and forecasting metasubject educational results. The FSES does not give a direct definition of metasubject results; the content of this term is revealed through listing of various skills and abilities, development of which indicates the achievement of a metasubject educational result. Based on several definitions highlighted in special literature, we can formulate a general definition of a metasubject educational result adopted in various fields related to the science of learning. Meta-subject educational results are “patterns of thinking, feeling, and behavior” that develop throughout an individual’s life and take some part in the learning process. More generally, these results may include personality characteristics that are not manifested in the form of subject educational results, but are related to socio-emotional or behavioral http://www.iaeme.com/IJCIET/index.asp 1650 [email protected] Arseniy Aleksandrovich Lebedev manifestations of an individual in the learning process and are the object of development in a school or influencing the formation of subject educational results. The above given definition is too abstract for purposes of this study. This sets the task of finding a working definition of a metasubject educational result that could be applied in connection with the use of recurrent neural networks to predict educational results. Such a concept should meet following important requirements: • it should be as close as possible to the notion of behavior, since it is the behavior that is the basic level with which algorithms that operate on data accumulated in SSR can work; • it should be clear to a teacher, since it is the teacher who is the main expert in assessment of metasubject educational results (only a teacher can be the source of empirical data that can be used to train and verify the recurrent neural network); • it should not contradict traditional definitions of a metasubject educational result, the general form of which is given above. An important step in the present study is the fundamental postulation of discreteness of metasubject educational results, numerical or categorical value of which could serve both as an input data of the recurrent neural network and the predicted value. A survey among teachers conducted in the course of a study on the GlobalLab platform showed that respondents refer sceptically to the need for a separate assessment of such metaobject educational results, such as, for example, self-control when reading or drawing up a development plan for mastering material. Many of respondents indicate that the presence of these educational results correlates well with subject educational results since they are a prerequisite for successful preparation for traditional forms of assessment. Some of respondents express a suggestion that these skills are missing as such. In their opinion, they are the part of a general cognitive process aimed at mastering the material. It should be noted that selection of individual skills (their discretization) is a technique that facilitates the study of cognitive processes (or one holistic process) occurring in mind of a learner. It is reasonable to assume that discretization of different metasubject educational results is based on two factors: • differences in behavior; • differences in behavior result (success/failure). We illustrate this assumption with the example of the metasubject result “self-control during reading”. The result of reading is assimilation of the read material, which can be verified by such traditional methods as retelling and answering a question about the text. When reading, a student may experience following variations in behavior leading to a successful result: 1) having an excellent memory, the reader sequentially moves from paragraph to paragraph without changing the speed of reading; 2) having an average memory, the reader “recounts” each read paragraph to himself; 3) the reader goes through all paragraphs and then returns to places that he did not remember or understand. In the first variation in reader's behavior there are no signs of controlling the reading process, and in other two variations, the reader makes efforts to control the reading process. All of them can lead to a successful result – a student will be able to retell the text and answer questions about it. It is differences in behavior in variations (1), (2) and (3) that suggest that in cognitive process in this case, variants of its course can be distinguished, which can be called the educational meta-object result "regulation of reading" for variations (2) and (3) Successful http://www.iaeme.com/IJCIET/index.asp 1651 [email protected] Methods of Measurement, Evaluation and Prediction of Metasubject Educational Results behavior is an important prerequisite for discretization of metasubject educational results. The situation, in which the reader has read entire text, re-read some parts of it and still could not complete tasks, is easily modeled. Consequently, whatever version of deep cognitive process in this case was – even if it is very close to a successful behavioral manifestation – it cannot be recognized as the indicating achievement of a metasubject educational result. The cognitive process, which is at a deep level, is not available for direct observation and assessment (including methods of machine learning). It manifests itself at the level of behavior, which is the basic level at which observation becomes possible. At this level, various variations of behavior can be distinguished, which in turn manifest themselves at a higher level – the level of result of behavior, which is rated as successful or unsuccessful one. It should be noted that such a traditional form of automated assessment of KSA as testing deals only with highest level – the result level. While the form of assessment in the form of an oral exam is aimed largely at study of a deeper level – the level of behavior. The question that this behavior and its variations in terms of their assessment is the core to the subsequent successful development of a mathematical model of use of a recurrent network for predicting educational results. This is due to the fact that the input data of network is data about behavior and its variations. This study uses simplified models of metasubject educational results. This approach is productive in terms of solving forecasting automation problems. Traditionally, the behavior is defined as a set of actions and their characteristics in a person, an animal, a system, or an artificial object that manifests itself in relation to itself or an environment and is a response to external stimuli. For purposes of this study, such an understanding of behavior needs clarification and simplification. This study focuses solely on the behavior as the part of learning process or evaluation of KSA. The behavior quanta are actions characterized by certain parameters and arranged in a certain sequence. In this case, the sequence of actions is no less important than their parameters, since it is precisely consideration of the sequence of events, that is, the main distinguishing feature of recurrent neural networks. Since variation of behavior plays an important role in discretization of metasubject educational results, the question of identity of behavior becomes extremely important: what variations should be considered identical or similar ones. Along with selection of variations in behavior, it seems appropriate to propose a working concept of a behavior invariant. A behavior invariant is an abstract view of common features that unite a certain set of variations. The approach in which specific metasubject result achieved with this variation is recognized as an invariant of behavior variations grounded within the framework of this study. As an illustration of variation in behavior and its invariant, Table 2 shows several examples of successful formulations of behavior invariants with their corresponding variations. http://www.iaeme.com/IJCIET/index.asp 1652 [email protected] Arseniy Aleksandrovich Lebedev Table 2. Examples of behavioral invariants (metasubject educational result) and their corresponding variations Invariant Variations 1. Expressing doubts about reliability of fact. 2. Formulation of a hypothetical alternative version of fact. 3. Search for information about a hypothetical version of the fact on Internet. 4. Analysis of information found. 5. Repeat steps 2 and 3. A critical analysis of fact 1. Expressing doubts about reliability of fact. 2. Search for alternative versions of the fact on Internet. 3. Analysis of information found. 1. Expressing doubts about the reliability of the fact. 2. Attracting knowledge already available to a student to refute/confirm the fact. 1. Decomposition of an object or a process under study. 2. Assignment to parts of the legend selected during decomposition. Creating models of objects and processes 3. Drawing up a general scheme of an object or process consisting of symbols of their parts. under study by presenting information in a character-symbol form 1. Creating a simplified representation of an object or a process as a diagram. 2. Identify duplicate fragments in a scheme. 3. Entering of a legend for repeating fragments. 4. Creating the final version of model with the legend. 1. Retelling of the just heard point of view of interlocutor. 2. The requirement for interlocutor to confirm correctness of interpretation of his point of view. 3. Expressing your own point of view. 4. Bringing evidence of loyalty to own point of view. 5. Taking illustrations confirming correctness of own point of view as an example. The argument of own point of view in 4. Highlighting differences from the point of view of response to hearing of the point of view of interlocutor. interlocutor 5. Conclusion. 1. Nomination of counterexamples that contradict the point of view of interlocutor. 2. Formulation of their point of view based on counterexamples. 3. Highlighting differences from the point of view of interlocutor. 4. Conclusion. http://www.iaeme.com/IJCIET/index.asp 1653 [email protected] Methods of Measurement, Evaluation and Prediction of Metasubject Educational Results A rather difficult question for predictive model of metasubject educational results to be built is the question of criteria for recognizing one or another variation of behavior as successful one. The concept of success in learning activities is traditionally associated with subject educational results and, in particular, with specific results obtained using various assessment methods (testing, oral examination, control paper, etc.). This understanding of success is also relevant for evaluation of metasubject educational results. However, in the present study it is planned to add new criteria formulated under consideration of complex project activities conducted by users on the GlobalLab platform to traditional success criteria (for example, estimates from electronic diary). Here are some examples of such possible success criteria: 1. User action – placing project on the GlobalLab platform; possible success criteria: approval of the project by the moderator, participation in the project of at least n other users, appearance of draft messages from other users on the discussion board. 2. User action – posting a message on the project discussion board; possible success criteria: receiving a reply to the message, achieving n-levels of inclusion of answers below the user's message. 3. User action – placing the project idea, possible success criteria: creation of a project by another user based on the idea, receiving a “like” by the idea from another user. As mentioned above, a metasubject educational result is available for observing by the researcher only in the form of certain variations of behavior. Behavioral variations, which are a peculiar manifestation, the “surface structure” of a real cognitive process, occur in mind of a learner and are not available for direct analysis. This situation is in many ways comparable to the situation in linguistics, which deals exclusively with surface structures and is forced to build rather complex hypothetical models reflecting deep structure of sentences. The analogy with language behavior of a person can be continued in the following aspect. Like the surface structure of an utterance, variation in behavior unfolds consistently over time and has a linear and discrete character, while the cognitive process (like forming deep structure of utterance) may have a non-linear and non-discrete nature. It should be particularly noted that analogy with linguistic problems in this case indirectly indicates possible reasons for successful use of recurrent networks for predicting educational results. Recurrent networks have been very successfully applied in processing and synthesis of texts in natural languages. 5. CONCLUSION An external observer who seeks to identify the presence or absence of a metasubject educational result actually has the access to all aspects of student’s behavior, but in many cases they are analyzed intuitively. This was reported by many teachers participating in the survey on the GlobalLab platform. Only in some cases an observer can accurately describe, on the basis of what signs in student's behavior he was able to conclude about absence/presence of a metasubject educational result. This feature is the main reason for numerous difficulties in formalization of procedures for measuring and evaluating metasubject educational results. From this point of view, the SSR has two important multidirectional features: • rich means of accurate recordation of almost all user actions therein; • almost complete absence of any means of monitoring the cognitive process. On one hand, these features expand possibilities of very accurate fixation of actions performed by a user and components of behavior variation, and on the other hand, they impose significant restrictions on completeness of manifestation of behavior variations in SSR. http://www.iaeme.com/IJCIET/index.asp 1654 [email protected] Arseniy Aleksandrovich Lebedev Manifestation of behavior variation at the level of SSR is composed of a large number of uniform user actions, which are recorded in SSR as a set of its “physical” actions (scrolling, mouse movements, etc.). Moreover, not all elements of behavior variations are expressed in SSR. It can be assumed that manifestation of the level of SSR is as poorly represented by the observed behavior variation as a behavior variation itself weakly represents the cognitive process occurring in student's mind. Thus, in SSR, deep signs of a cognitive process characterized by the presence or absence of a metasubject educational result, pass through two filters that sort out some of these signs – through the observed behavior variation (filter 1) and its manifestation in SSR (filter 2). The fact that after the first filter there are signs sufficient to distinguish metasubject educational result, is confirmed by fact that an experienced expert who carefully monitors a student, can measure and evaluate this result (via recordation and questioning TAP, MAI, MSLQ described above). The high correlation of results of diagnostics when applying the Otter SSR of the Veenmann group with diagnostics using TAP recordation method (0.96) confirms that the second filter also misses a number of signs sufficient to detect a meta-subject educational result. This prediction is also supported by the accuracy of forecasting in papers in which the subject educational result is predicted solely on the basis of data on metasubject: Prediction accuracy ranges from 69 to 83%. FUNDING STATEMENT Applied research described in this paper is carried out with financial support of the state represented by the Russian Federation Ministry for Education and Science under the Agreement #14.576.21.0091 of 26 September 2017 (unique identifier of applied research RFMEFI57617X0091). REFERENCES [1] [2] [3] [4] [5] Altinay, F., Dagli, G. and Altinay, Z.. Role of Technology and Management in Tolerance and Reconciliation Education. Quality & Quantity, 51(6), 2017, pp. 2725–36. https://doi.org/10.1007/s11135-016-0419-x. Carroll, N., Richardson, I. Moloney, M. and O’Reilly, P. Correction to: Bridging Healthcare Education and Technology Solution Development through Experiential Innovation. Health and Technology, 8(4), 2018, pp. 263. https://doi.org/10.1007/s12553017-0215-1. Fernandes, R., Geraldo, W., Rodrigues, A. M. and Ferreira, C. A. Conceptions of the Nature of Science and Technology: A Study with Children and Youths in a Non-Formal Science and Technology Education Setting.” Research in Science Education 48(5), 2018, pp. 1071– 1106. https://doi.org/10.1007/s11165-016-9599-6. DeCoito, I., Steele, A. and Goodnough, K. Introduction to the Special Issue on Science, Technology, Engineering, and Mathematics (STEM) Education. Canadian Journal of Science, Mathematics and Technology Education, 16(2), 2016, pp. 109–13. https://doi.org/10.1080/14926156.2016.1166298. Polly, D. and Rock, T. Elementary Education Teacher Candidates’ Integration of Technology in the Design of Interdisciplinary Units. TechTrends, 60(4), 2016, pp. 336–43. https://doi.org/10.1007/s11528-016-0059-y. http://www.iaeme.com/IJCIET/index.asp 1655 [email protected] Methods of Measurement, Evaluation and Prediction of Metasubject Educational Results [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] Abramov, R. A., Tronin, S. A., Brovkin, A. V., Pak, K. C. Regional features of energy resources extraction in eastern Siberia and the far east. International Journal of Energy Economics and Policy, 8(4), 2018, pp. 280–287. Cervera, M. G., and Johnson, L. Education and Technology: New Learning Environments from a Transformative Perspective. International Journal of Educational Technology in Higher Education, 12(2), 2015, pp. 1–13. https://doi.org/10.7238/rusc.v12i2.2570. Gallardo Echenique, E., Molías, L. M. and Bullen, M. Students in Higher Education: Social and Academic Uses of Digital Technology. International Journal of Educational Technology in Higher Education, 12(1), 2015, pp. 25–37. https://doi.org/10.7238/rusc.v12i1.2078. Memmedova, K. and Selahattin, E. L. Effects of the Technology Use on Anxiety and Aggression Levels of Youth Conducting Their Higher Education Studies Abroad. Quality & Quantity, 52(1): 2018, pp. 501–7. https://doi.org/10.1007/s11135-017-0630-4. Abramov R. A. Management Functions of Integrative Formations of Differentiated Nature. Biosci Biotech Res Asia, 12(1), 2015, pp. 991–997. Henriksen, D. et al. Creativity and Technology in Education: An International Perspective. Technology, Knowledge and Learning, 23(3), 2018, pp. 409–24. https://doi.org/10.1007/s10758-018-9380-1. Langub, L. W., and Lokey-Vega, A. Rethinking Instructional Technology to Improve Pedagogy for Digital Literacy: A Design Case in a Graduate Early Childhood Education Course. TechTrends, 61(4), 2017, pp. 322–30. https://doi.org/10.1007/s11528-017-0185-1. Fletcher-Watson, S. Evidence-Based Technology Design and Commercialisation: Recommendations Derived from Research in Education and Autism. TechTrends, 59(1), 2015, pp. 84–88. https://doi.org/10.1007/s11528-014-0825-7. Hodson, D. Giving Canadian Science, Mathematics, and Technology Education an Independent Voice. Canadian Journal of Science, Mathematics and Technology Education, 15(4), 2015, pp. 339–43. https://doi.org/10.1080/14926156.2015.1091901. Williams, P. J. Research in Technology Education: Looking Back to Move Forward …again. International Journal of Technology and Design Education, 26(2), 2016, pp. 149– 57. https://doi.org/10.1007/s10798-015-9316-1. Brenner, A. M. and Brill, J. M. Investigating Practices in Teacher Education That Promote and Inhibit Technology Integration Transfer in Early Career Teachers. TechTrends, 60(2), 2016, pp. 136–44. https://doi.org/10.1007/s11528-016-0025-8. Elshof, L. What Is a Canadian Technology Education? Questions of Distinction and Sustainability. Canadian Journal of Science, Mathematics and Technology Education, 15(4), 2015, pp. 418–29. https://doi.org/10.1080/14926156.2015.1091903. Tsai, Ch.-W. The Use of Mobile Technology and Ubiquitous Computing for Universal Access in Online Education. Universal Access in the Information Society, 15(3), 2016, pp. 313–14. https://doi.org/10.1007/s10209-014-0389-5. Ørberg, J. W. Uncomfortable Encounters between Elite and ``shadow Education’’ in India--Indian Institutes of Technology and the Joint Entrance Examination Coaching Industry. Higher Education, 76(1), 2018, pp. 129–44. https://doi.org/10.1007/s10734-017-0202-5. Duart, J. M., and James, R. New Era and Challenges for a Consolidated Journal in Educational Technology in Higher Education. International Journal of Educational Technology in Higher Education, 13(1), 2016, pp. 1. https://doi.org/10.1186/s41239-0160001-6. http://www.iaeme.com/IJCIET/index.asp 1656 [email protected]