A TAXONOMY FOR EVALUATING INFORMATION CREDIBILITY ON THE INTERNET

A TAXONOMY FOR EVALUATING INFORMATION CREDIBILITY ON THE INTERNET Submitted in (partial) fulfilment of the requirements for the degree of BACHELOR OF SOCIAL SCIENCE (HONOURS) IN INFORMATION SYSTEMS (FACULTY OF HUMANITIES) RHODES UNIVERSITY by HAKAN TERZI AUGUST 2016 A TAXONOMY FOR EVALUATING INFORMATION CREDIBILITY ON THE INTERNET by Hakan Terzi SUPERVISOR/S: ED DE LA REY DEPARTMENT: DEPARTMENT OF INFORMATION SYSTEMS FACULTY: FACULTY OF HUMANITIES, RHODES UNIVERSITY DEGREE: HONOURS ABSTRACT The rise of the internet has led it to become the biggest information sharing platform in the world today. Its ease of access and users’ ability to upload anything they see fit means that it is a benefit as well as a disadvantage. It has resulted in the amount of quality information disseminated being reduced through the two concepts of misinformation and disinformation. These phenomena may have detrimental effects on society and compromise the credibility/trustworthiness of a vast number of online information sources. It is therefore important for internet users to evaluate sources that may influence them or their knowledge of the world in any way. This paper aims to provide internet users with a framework/taxonomy that may be used in an effort to access quality information. It achieves this through the provision of a scoring system that evaluates each source based on a number of certain criteria. This provides the evaluator with a total score which may be placed in one of the three categories of information, misinformation or disinformation. These categories indicate the possible nature of the source and whether they should be trusted. KEYWORDS: Information, misinformation, disinformation, credibility ii Declaration I declare that the Dissertation/Thesis entitled, A Taxonomy for Evaluating Information Credibility on the Internet, which I hereby submit for the degree, Bachelor of Social Science (Honours) at Rhodes University, is my own work. I also declare that this thesis/dissertation has not previously been submitted by me for a degree at this or any other tertiary institution and that all the sources that I have used or quoted have been indicated and acknowledged by means of complete references. ________________ Hakan Terzi (signed) iii Acknowledgements I wish to thank my supervisor, Prof Ed de la Rey for the way in which he empowered me to do and complete my research. I thank him for his support and guidance in setting me on the path that led me to producing the resultant proposed framework. I would also like to thank Emmalene Le Roux for proof reading my work as well as providing me with sustenance during my hours spent in the lab. iv Table of Contents CHAPTER 1 .............................................................................................................................. 1 Introduction and Overview ........................................................................................................ 1 1.1 Research Background .................................................................................................. 1 1.2 Problem Statement ...................................................................................................... 3 1.3 Research Objectives .................................................................................................... 4 1.4 Research Questions ..................................................................................................... 4 1.5 Scope of Research ....................................................................................................... 4 1.6 Significance of Research ............................................................................................. 5 1.7 Methods ....................................................................................................................... 5 1.8 Ethical Considerations................................................................................................. 6 CHAPTER 2 .............................................................................................................................. 7 Literature Review....................................................................................................................... 7 2.1 Introduction ................................................................................................................. 7 2.2 Information, Misinformation and Disinformation ...................................................... 8 2.2.1 Information .......................................................................................................... 8 2.2.2 Information and Truth ........................................................................................ 10 2.2.3 Misinformation and Disinformation .................................................................. 10 2.3 Varieties of Disinformation ....................................................................................... 12 2.4 Credibility.................................................................................................................. 13 2.4.1 Credibility in Passive Informing ........................................................................ 13 2.4.2 Credibility in Active Informing ......................................................................... 13 2.4.3 Credibility on the Internet .................................................................................. 14 2.5 Misinformation and Disinformation on the Internet ................................................. 14 2.6 Evaluation of information on the internet ................................................................. 16 Misinformation Prevention Framework ........................................................................... 18 2.7 Conclusion................................................................................................................. 21 CHAPTER 3 ............................................................................................................................ 23 Current Frameworks ................................................................................................................ 23 3.1 Introduction ............................................................................................................... 23 3.2 Discussion ................................................................................................................. 23 3.2.1 Credibility .......................................................................................................... 24 3.2.2 Credibility Criteria ............................................................................................. 25 v 3.2.3 3.3 Misinformation Prevention Framework ............................................................. 26 Conclusion................................................................................................................. 27 CHAPTER 4 ............................................................................................................................ 29 Proposed Framework ............................................................................................................... 29 4.1 Introduction ............................................................................................................... 29 4.2 Details of Proposed Framework ................................................................................ 29 4.3 How it Works ................................................................................................................. 34 4.4 Conclusion ..................................................................................................................... 35 CHAPTER 5 ............................................................................................................................ 36 Discussion ................................................................................................................................ 36 5.1 Introduction ............................................................................................................... 36 5.2 Interpretation of Proposed Framework ..................................................................... 36 5.2.1 Case Study 1 ...................................................................................................... 36 5.2.2 Case Study 2 ...................................................................................................... 39 5.3 Conclusion ..................................................................................................................... 42 CHAPTER 6 ............................................................................................................................ 43 Conclusion and Future Work ................................................................................................... 43 6.1 Introduction .................................................................................................................... 43 6.2 Final Summary and Conclusion ..................................................................................... 43 6.3 Future Work ................................................................................................................... 44 References ................................................................................................................................ 45 Appendices ............................................................................................................................... 48 vi List of Figures Figure 1: Misinformation Prevention Framework (Koohang and Weiss, 2003) ........... 19 Figure 2: Proposed Framework .......................................................................................... 28 Figure 3: Misinformation Prevention Framework Adaptation ......................................... 30 Figure 4: Proposed Information Evaluation Framework ................................................. 34 Figure 5: Scoresheet – Case Study 1 ................................................................................ 37 Figure 6: Scoresheet – Case Study 2 ................................................................................ 40 Figure 7: Blank Scoresheet ................................................................................................. 48 List of Tables Table 1: A summary of the features of information, misinformation and disinformation (Karlova and Fisher, 2012) .................................................................................................. 11 Table 2: Relationship between the 5 criteria for the evaluation of information on the internet (Tudjman and Mikelic, 2003)................................................................................. 18 Table 3: Most Prominent Credibility Criteria (Tudjman and Mikelic, 2003) .................. 25 Table 4: Information Sources and Related Scores .......................................................... 31 Table 5: Relationship between credibility criteria for the evaluation of information on the internet with corresponding scores .............................................................................. 33 Table 6: Total Score and Probable Type of Information Source ................................... 35 vii CHAPTER 1 Introduction and Overview 1.1 Research Background In the modern world we live in, a multitude of information is being thrown at us from all directions and it is becoming increasingly important to go back to the basics and discuss the quality of the information that we are receiving and processing. Quality information possesses the traits of accuracy, timeliness, relevance, as well as understandability of the content (Hernon, 1995). If the information we receive lacks any of these particular traits, then the quality of the information is compromised. Information quality is compromised through two separate concepts known as ‘misinformation’ and ‘disinformation’. Although both are similar in the sense that they result in information quality being compromised, they do differ greatly as will be made clear through this proposal which will lead me to describing and unfolding the latter concept of disinformation. It may be difficult to judge the quality of disinformation, however it could be viewed in a similar manner as deceptive lying (Fetzer, 2004). Although some forms of lying have no intention to deceive, for example a comedian’s jokes, the intention of disinformation is to use lying as a means to carry out a certain, sometimes sinister objective (Fetzer, 2004). According to Stahl (2006), misinformation can be described as information that is “wrong or misleading”, whereas disinformation is described as the “deliberate dissemination of false information”. This means that people providing disinformation must have an intent to deceive their audience. The definitions of the two concepts used above illustrates that disinformation is a subset of misinformation. That is, disinformation will always entail misinformation. Misinformation on the other hand does not necessarily entail disinformation. This is owing to the fact that a source eliciting misinformation may do so purely by mistake and have no intention of deceiving their audience. A misinformation source may themselves believe that the information they are giving out to be truthful and accurate. 1 Although some definitions seem to point to disinformation as being a subset of misinformation (as described above), several arguments have also been put forward stating that both misinformation and disinformation are extensions of information. That is, they are both separate sub-categories of information (Karlova and Fisher, 2012). Fox (1983) indicated that misinformation (albeit false) is a ‘species’ of information and that information may still be obtained from misinformation, making it informative. Similarly, according to Fallis (2009), disinformation “will typically be inaccurate. It does not have to be inaccurate. It just has to be misleading.” Therefore, due to the fact that misinformation can be untrue and disinformation can be accurate, both concepts are separate, yet equal sub-categories of information (Karlova and Fisher, 2012). Karlova and Lee (2011) note that misinformation and disinformation discussions tend to suffer from imprecise term descriptions owing to a lack of definitions, theories and research. In addition, these terms are used in various other fields such as computer science, psychology and philosophy. This may be the reason why there is conflict between researchers as to how the two terms contrast and relate to information. With the internet being the most common source of information nowadays, it has naturally become the biggest highway for distributing disinformation (Fallis, 2009). Disinformation has been present in various forms even prior to the internet age more discretely through the likes of government propaganda, false advertising, forged documents, falsified maps and photos (Fallis, 2009). However, the information technologies of today are making it relatively simple for people to deceive others through the distribution of disinformation. That being said, the various forms of disinformation that are presented above are more prominent today owing to people’s access to and reliance on the internet which makes it much easier to obtain and be exposed to disinformation. Added to the various forms of disinformation above are the likes of bogus websites, internet fraud and even Wikipedia entries (Fallis, 2014). Examples of these include disinformation on topics such as investment opportunities, stock markets, politics, news propaganda, social media and medical treatments. As one can see, these are important topic areas and a person/group of people affected by this type of disinformation may be caused significant harm. 2 1.2 Problem Statement There is a lack of a standard method and procedure when it comes to establishing the nature of information sources on the internet (Keshavarz, 2016). Significant problems may arise for users of the internet due to the fact that they are continuously exposed to information that is both misleading and inaccurate, and furthermore the ignorance of said users, as people tend to believe and trust everything they see and hear on the world wide web (Karlova and Fisher, 2012). Disinformation is present in a great deal of places on the internet and poses an enormous threat to the integrity of the information that we receive through it. The popularity and spread of the internet has made it more accessible than it has ever been which brings with it an increased danger of people being exposed to disinformation (Keshavarz, 2016). Sources of disinformation have also become increasingly attracted to spreading their deceit through the internet due to its popularity and ease of access. For example, Twitter is an extremely popular social media tool that is used by millions of people to communicate and share their views. This makes Twitter a powerful instrument for the distribution of information which therefore makes it an equally powerful instrument for the distribution of disinformation (Chamberlain, 2009). Chamberlain (2009) states that Twitter’s casual nature of communication makes it a particularly suitable means for the dissemination of disinformation. Communication through Twitter, be it from a single person or a large organisation, requires insignificant resources, making it a simple method to distribute information which very often disinforms. Empirical studies have been carried out in an attempt to determine the magnitude of inaccurate information on the internet, however it is difficult to specify how much of this misinformation is intentionally misleading (Fallis, 2009). This is because it is difficult to identify the agenda of the sources of this misinformation/disinformation. It is therefore important to develop framework or taxonomy that will attempt to tackle the degradation of information quality distributed to the masses via the internet. 3 1.3 Research Objectives The objective of my research is to ascertain the nature of the different types of information present online in order to better identify and possibly tackle the ‘problem’ of disinformation on the internet. Through my research I will develop a taxonomy for evaluating information in order to identify sources of misinformation and disinformation by using the works of various other authors in the field. This taxonomy will be used as a lens in an attempt to categorize the nature of different sources of information on the web. This will optimistically uncover the amount of disinformation that exists on the world wide web and the bias of certain sources towards their own agendas. 1.4 Research Questions   How does one distinguish between information, misinformation and disinformation?  internet? What frameworks are available to identify misinformation and disinformation on the How is the credibility of information on the internet evaluated? 1.5 Scope of Research The aim of this research is to identify different types and nature of information sources on the internet. That is, material on their websites, Facebook pages, Twitter handles and the like. The taxonomy that is developed will only be of use on the internet as it is the specific reason the taxonomy is being developed. This means that one will not be able to use this taxonomy in any other fields such as religion, scientific papers and television. My research also assumes that the existing similar frameworks that I use to create my own taxonomy/framework are accurate and true. This is important because if the frameworks I use to incorporate into my own are later shown to be inaccurate and wrong, it will delegitimise my own research due to the fact that I developed it on false information. 4 1.6 Significance of Research There is a vast amount of information from past research (that will be made clear in my literature review) illustrating that information sources do partake in disinformation as a means to influence others. This may be carried out through various channels which ties into my research. This research is important because it may illustrate how easy it is for information sources to deceive their audience. This could have great implications for internet users in future because most people view the information they read on the internet to be impartial and unbiased. This research, if successful, may help people become aware of the fact that information sources at times misinform or attempt to disinform the public based on their own agendas in order to swing public opinion. If this research develops a suitable framework for evaluating information credibility on the internet, then it may set the stage for further research to be carried out in this field. 1.7 Methods I will begin my research by defining the terms information and information quality. This is an important first step because it is the foundation for differentiating the concepts of misinformation and disinformation. I will then illustrate a detailed description of the two terms, making the differences between the two concepts clear. Disinformation is the focus area of my research which will lead me to presenting several theories and models describing the concept. Contrasting views of disinformation will be made evident which will illustrate the conflict and confusion around a suitable definition and model of the term at this point in time. I will examine the attributes describing information credibility put forward by different authors and strategically select the attributes that best fit my research, as well as the similarities between these various models. This will help me produce my own taxonomy that will become a lens through which the quality of information found on the internet may be evaluated. Ultimately I will be using the developed taxonomy to examine, critique and rate different online information sources. To do this, my developed taxonomy will have different levels/ranks of information and internet sources will be ranked based on the created taxonomy. 5 1.8 Ethical Considerations Due to the fact that this is a theoretical research paper, there are very few, if any, ethical considerations to be made. 6 CHAPTER 2 Literature Review 2.1 Introduction The rise of the internet has led it to become the biggest information sharing platform in the world today. The ease of access to the internet in addition to the freedom it provides its users makes it a benefit as well as a hazard (Koohang and Weiss, 2003). Its benefit is that it provides users with an incomprehensible amount of information from all over the world that is made accessible through any device capable of accessing it as well as in internet connection. However, finding information that is of a high quality has become an increasingly important task due to the rise of untrustworthy information being posted to the internet (Keshavarz, 2014). Untrustworthy information presents itself in the forms of misinformation and disinformation. This makes identifying and assessing the information quality of internet resources a vital task in ensuring the information that one accesses is indeed truthful (Keshavarz, 2014). The term disinformation has been around for approximately fifty years which makes it a relatively new term compared to other terms related to this research topic such as ‘information’ and ‘lying’ (Fallis, 2009). This has resulted in the definition of ‘disinformation’ not being ‘fixed’ due to numerous authors putting their own definitions of the term forward (Fallis, 2009). However, in order to completely understand this topic and effectively communicate with one another, a wholly ‘stable’ definition of the word must be shared by all parties (Fallis, 2009). This will be achieved through this literature review by reviewing past descriptions of the term in order to identify commonalities between definitions and put forward our own understanding of the term ‘disinformation’ which will be used to describe the term going forward in this research paper. This paper will begin by describing the foundations of disinformation which is ‘information’, including a few of its characteristics. It will then describe misinformation and disinformation followed by a summary of the differences between the three terms. Once this is done we will then be able to relate these phenomena specifically to the internet and describe measures 7 through which we can evaluate information and identify misinformation and disinformation on the internet. 2.2 Information, Misinformation and Disinformation Before one can investigate the concept of disinformation, it is imperative that one understands the building blocks of the term. These building blocks include terms such as ‘information quality’, ‘truth’ and ‘credibility’, as well as terms similar to disinformation such as ‘misinformation’. Once the research has made clear the definitions and differences of the above terms, it will be possible to examine and explore the concept of disinformation, and more specifically, disinformation on the internet. This may sound simple enough; however, due to the fact that the three concepts (information, misinformation and disinformation) are closely related to one another, it may pose a difficulty when searching for the truest understanding of disinformation (Kumar and Geethakumari, 2014). 2.2.1 Information The Oxford Dictionary simply defines the term ‘information’ as “facts provided or learned about something or someone”. This is a basic definition of the word because in essence, the term ‘information’ today has a number of different connotations due to the many different fields and sciences that it is used in (Koohang and Weiss, 2003). This paper’s field of research is ultimately disinformation on the internet which may be placed in the field of information science. Rafael Cappuro has been at the forefront of information science research for several decades and is a prominent theoretician in the field (Tudjman and Mikelic, 2003). Cappuro’s paper, The Foundations of Information Science concludes with the following statement: “Information science, conceived as a hermeneutic-rhetorical discipline, studies the con-textual pragmatic dimensions within which knowledge is shared positively as information and negatively as misinformation particularly through technical forms of communication” (Cappuro, 2000). Stvilia et al. (2006) identify quality of information as one of the key determinants of the quality of decisions and actions made by individual and institutional practices that are dependent on information. Owing to the volume and diversity of information available, producers of 8 information have grown at an unmanageable rate which is making it increasingly difficult to identify information that is of a poor quality (Stvilia et al., 2006). In order for information to be described as ‘quality information’, thus making it trustworthy and reliable, it must possess the traits of accuracy, timeliness, relevance, as well as understandability of the content (Hernon, 1995). Stairs and Reynolds (2013) added to this list and noted that quality information must also be complete, economical, reliable, secure, verifiable and flexible. If the information we receive lacks any of these particular traits, then the quality of the information is compromised. As mentioned above, the term information is comprised of facts, and facts are those things recognised or attested to be true. When looking at the list of traits however, ‘truthfulness’ does not appear, leading us to query; is it necessary for the information to be true to be considered information? (Koohang and Weiss, 2003) For example, if information lacks accuracy, it may lead to people acquiring skewed beliefs about the world around them (Fallis, 2009). Information that is inaccurate may be misleading and can have bad consequences irrespective of whether the source of the information made an honest mistake (i.e. Misinformation) or intentionally intended to deceive their audience (i.e. Disinformation) (Fallis, 2009). Information quality is reduced through two separate concepts known as ‘misinformation’ and ‘disinformation’. Although both are similar in the sense that they result in information quality being reduced, they do differ greatly as will be made clear through this review which will lead me to describing and unfolding the latter concept of disinformation. It may be difficult to judge the quality of disinformation; however, it could be viewed in a similar manner as deceptive lying (Fetzer, 2004). Although some types of lying have no intention to deceive, for example a comedian’s jokes, the intention of disinformation is to use lying as a means to carry out a certain, sometimes sinister objective (Fetzer, 2004). When a person first hears the term disinformation, the association with information will logically be established. This stems from the fact that the prefix dis- implies privacy/negativity combined with the word information; providing the idea that an inconsistent set of data that negatively informs is being passed onto the audience (Oosterwoud, 2015). 9 2.2.2 Information and Truth It must be made clear that there is no universally accepted definition of ‘truth’ (Stahl, 2006). However, since we are constantly dealing with information from all around us, we are forced to use our own understanding of the term in order to come to our own conclusions. This process may result in confusion and misunderstandings of the world we live in (Stahl, 2006). Information that lacks ‘truth’ results in the production of misinformation and disinformation; however, as will be discussed below, there are times when misinformation and disinformation lack truth but may still provide information (Karlova and Fisher, 2012). We can therefore state that an important characteristic of information is that it must be true, otherwise it may not be classified as information (Stahl, 2006). 2.2.3 Misinformation and Disinformation Simply put, misinformation can be described as information that is ‘wrong or misleading’, whereas disinformation is described as the ‘deliberate dissemination of false information (Stahl, 2006; Fallis, 2014; Karlova and Fisher, 2012; Fetzer, 2004). This means that people providing disinformation must have an intention to deceive their audience, whereas misinformation is misleading information that is disseminated unintentionally (Koohang and Weiss, 2003). As with information, there are various contrasting views of the term disinformation as it has been defined by numerous authors since it was first conceived. For example, Floridi, an information scientist, has updated his initial definition of disinformation several times. His initial definition stated that “disinformation arises whenever the process of information is defective” (Floridi, 1996, 509). Whereas, his most recent definition states “Disinformation is simply misinformation purposefully conveyed to mislead the receiver into believing that it is information” (Floridi, 2011, 260). The definitions of the two concepts used above illustrates that disinformation is a subset of misinformation. That is, disinformation will always entail misinformation. Misinformation on the other hand does not necessarily entail disinformation. This is owing to the fact that a source eliciting misinformation may do so purely by mistake and have no intention of deceiving their audience. A misinformation source may themselves believe that the information they are giving out to be truthful and accurate (Fallis, 2011). 10 Although some definitions seem to point to disinformation as being a subset of misinformation (as described above), several arguments have also been put forward stating that both misinformation and disinformation are extensions of information. That is, they are both separate sub-categories of information (Karlova and Fisher, 2012). Fox (1983) indicated that misinformation (albeit false) is a ‘species’ of information and that information may still be obtained from misinformation, making it informative. Similarly, according to Fallis (2009), disinformation “will typically be inaccurate. It does not have to be inaccurate. It just has to be misleading.” Therefore, due to the fact that misinformation can be untrue and disinformation can be accurate, both concepts are separate, yet equal sub-categories of information (Karlova and Fisher, 2012). Below is a table summarizing the features and differences between information, misinformation and disinformation: Information Misinformation Disinformation Y Y/N Y/N Y/N Y/N Y/N Current Y Y/N Y/N Informative Y Y Y/N Deceptive N N Y True Complete Y = Yes; N = No; Y/N = Could be Yes or No, depending on context & time Table 1: A summary of the features of information, misinformation and disinformation (Karlova and Fisher, 2012) Karlova and Lee (2011) note that misinformation and disinformation discussions tend to suffer from imprecise term descriptions owing to a lack of definitions, theories and research. In addition, these terms are used in various other fields such as computer science, psychology and philosophy. This may be the reason why there is conflict between researchers as to how the two terms contrast and relate to information. 11 2.3 Varieties of Disinformation Don Fallis’ paper, ‘A Conceptual Analysis of Disinformation’ (2009) describes a selection of the main varieties of disinformation that he has identified through his research. The different varieties are as follows: i. Disinformation is regularly thought to be a governmental or military activity. Fallis (2009) points out the fact that the standard dictionary definition of the term disinformation is “deliberately misleading information announced publicly or leaked by a government or especially an intelligence agency” (Oxford English Dictionary, 2016). Additionally, news services are also often a typical source of disinformation (Fallis, 2009). ii. Disinformation is frequently the result of a technically sophisticated and carefully planned deception. An example of this could be hackers altering news services’ websites and intentionally disseminating misinformation (Fallis, 2009). iii. Disinformation is not always disseminated directly from the individual or organisation that is intending to deceive. For example, news services are at times fooled into distributing misleading or inaccurate information that is generated by someone else (Fallis, 2009). iv. Disinformation is most commonly verbal or written information but is sometimes represented by other means such as doctored photographs and falsified maps (Fallis, 2009). v. There is often a very wide distribution of disinformation from its source. For example subscribers of a newspaper and anyone with an internet connection or a television (Fallis, 2009). vi. The intended audience/victim of disinformation is typically a person or a group of people; however, there are some cases where disinformation may be aimed at a machine. An example of this could be a case where website managers attempt to ‘fool’ search engine indexes in an attempt to redirect traffic to their webpage (Fallis, 2009). 12 2.4 Credibility An indirect role in determining the quality of information is through the credibility of its content (Gackowski, 2006). The credibility of information simply means whether it is true. In other words, whether one may rely on the value of that information (Gackowski, 2006). In this regard, complete credibility is extremely difficult to achieve because the adjective ‘true’ implies consistency with reality. It may therefore be said that misinformation/disinformation sources misrepresent reality through the information they convey. True/valid information authentically represents reality (Gackowski, 2006). Sources of information content may be either passive or active. A passive information source will only yield information when observed, inspected or measured. On the other hand, an active information source is an informing entity that, by design or nature, transmits, disseminates, or broadcasts information. Examples of active information sources may be politicians, advertisers, preachers, etc. (Gackowski, 2006). 2.4.1 Credibility in Passive Informing Informing entities of a passive nature are actual objects and processes, some of which may be common and effortlessly recognised; others being utterly hidden to the naked eye of the entities informed (Gackowski, 2006). This means that the information should be well defined, objective (unbiased), relevant, accurate, of known variability, current (up to date) and precise. These traits are often attributed to the reputation of the source, which may be traced back to the originator and if the source is of recognised authorship (Gackowski, 2006). Furthermore, there is a positive relationship between the variety/number of independent sources and the credibility of the information value; that is, if there are a high number of independent sources that convey the same information, then there is a higher chance that the information conveyed is credible (Gackowski, 2006). 2.4.2 Credibility in Active Informing The above aspects of credibility in passive informing all apply to active informing. There is however a difference concerning ‘bias’ and disinformation in this case. Firstly, ‘bias’ may be defined as a “failure to fully inform” (Gackowski, 2006). In the case of passive informing, bias 13 is a result of ignorance, making it an aspect of misinformation. With active informing, there is a natural tendency of the information source to live towards bias and disinformation (Gackowski, 2006). Bias and disinformation are in fact the inescapable characteristics of active informing. With regard to active informing, bias and disinformation stem from the information source’s alignment of attitudes and interests, both of which may increase or decrease the reputation of the source (Gackowski, 2006). 2.4.3 Credibility on the Internet The rapid growth of the internet has led to the fast-changing nature of information resources available to us which in turn has resulted in the in the massive increase in the amount of untrustworthy information being disseminated on the internet. The abundance of different information available on the internet, as well as the lack of authorship (as compared to print material), has made it problematic determining whom and what to trust (Fisher et al., 2008). This makes evaluating the information quality of these resources a crucial task (Robins, Holmes and Stansbury, 2010; Keshavarz, 2014). Credibility has been a major consideration in many areas of research and practice such as health, commerce, and politics; yet despite the high significance of credibility of information on the internet, researchers have failed to give adequate attention to this field (Liu, 2004). Research that has been carried out has found that credibility is investigated most comprehensively at the website/structural level (Keshavarz, 2014). 2.5 Misinformation and Disinformation on the Internet With the internet being the most common source of information nowadays, it has naturally become the biggest highway for distributing misinformation and disinformation (Keshavarz, 2014). Various types of misinformation online include out-of-date information, incomplete information, factual errors, contradictions, improperly translated data, pranks, software incompatibilities and biased information (Keshavarz, 2014). Disinformation has been present in various forms even prior to the internet age more discretely through the likes of government propaganda, false advertising, forged documents, falsified maps and photos (Fallis, 2009). However, the information technologies of today are making it relatively simple for people to deceive others through the distribution of disinformation. That being said, the various forms of disinformation that are presented above are more prominent today owing to people’s access to and reliance on the internet which makes it much easier to obtain and be exposed to 14 disinformation (Fallis, 2009). Added to the various forms of disinformation above are the likes of bogus websites, internet fraud and even Wikipedia entries (Fallis, 2014). Examples of these include disinformation on topics such as investment opportunities, stock markets, politics, news propaganda, social media and medical treatments. As one can see, these are important topic areas and a person/group of people affected by this type of disinformation may be caused substantial harm. Significant problems may arise for users of the internet due to the fact that they are continuously exposed to information that is both misleading and inaccurate, and furthermore the ignorance of said users, as people tend to believe and trust everything they see and hear on the world wide web (Fallis, 2009). The three distinct problems that have been identified to cause misinformation and disinformation on the internet are: the nature of internet structure, the architecture of the internet, and the traditional causes of misinformation (Keshavarz, 2014). Problems associated with the architecture of the internet are the lack of central authority and data malleability. Traditional causes of misinformation are problems such as misconduct, human error, bias, and lack of currency and the removal of information from context (Keshavarz, 2014). The causes of disinformation are different because they are associated with the information source’s intent to falsify the information. These intentions may be political, instructional, malicious entertainment, or political (Keshavarz, 2014). Disinformation is present in a great deal of places on the internet and poses an enormous threat to the integrity of the information that we receive through it. The popularity and spread of the internet has made it more accessible than it has ever been which brings with it an increased danger of people being exposed to disinformation. Sources of disinformation have also become increasingly attracted to spreading their deceit through the internet due to its popularity and ease of access. For example, Twitter is an extremely popular social media tool that is used by millions of people to communicate and share their views. This makes Twitter a powerful instrument for the distribution of information which therefore makes it an equally powerful instrument for the distribution of disinformation. Chamberlain (2009) states that Twitter’s casual nature of communication makes it a particularly suitable means for the dissemination of disinformation. Communication through Twitter, whether it is from a single person or a large organisation, requires insignificant resources, making it a simple method to distribute information which very often disinforms. 15 Empirical studies have been carried out in an attempt to determine the magnitude of inaccurate information on the internet; however, it is difficult to specify how much of this misinformation is intentionally misleading (Fallis, 2009). This is because it is difficult to identify the agenda of the sources of this misinformation/disinformation. It is therefore important to develop frameworks that will attempt to tackle the degradation of information quality distributed to the masses via the internet. 2.6 Evaluation of information on the internet Distinguishing the three concepts of information, misinformation and disinformation on the internet may be made easier through the following criteria (Tudjman and Mikelic, 2003):  Informational websites provide the viewer with factual information. They provide evidence of the author as well as the source and owner of the information. They are upto-date and contain valid, accurate information that is without errors and subjectivity  (Tudjman and Mikelic, 2003). Websites containing misinformation result in confusion because they are misrepresented, contradictory to the previous information source or altered somewhere along the communication process. These websites may be out-of-date, contain subjective information, including errors. Although it may not be the intent of the information source, they may express views that are greatly subjective (Tudjman and  Mikelic, 2003). Disinformation websites have the intention of misleading and deceiving their audience. Clear signs of this may include a lack of author, no contact information, no reference of the information source and no date of when this ‘information’ was written (Tudjman and Mikelic, 2003). Two basic skills that internet users should be aware of are critical thinking and information literacy. These terms are very similar and are sometimes used interchangeably (Keshavarz, 2014). Weiler (2004) acknowledges that these two skills are required to effectively carry out a credibility assessment. The evaluation of content ties in closely with credibility assessment which in turn is an indicator of the information’s quality. (Keshavarz, 2014). Below are five criteria that have been put forward by various authors that may be used in the evaluation of content on the internet:  Authority 16   Accuracy  Currency  Objectivity Coverage/Scope (Tudjman and Mikelic, 2003; Keshavarz, 2014; Tate, 1999; Beck, 1997; Smith, 1997; Meola, 2004) There are numerous other criteria (such as accessibility, relevance, durability and privacy) that have been proposed by other researches; however, the above five criteria above share the most acceptance between the numerous authors. There is albeit, no set criteria for information evaluation among researchers (Keshavarz, 2014). Authority Accuracy Information Misinformation Disinformation Authors and sponsors are known; furthermore, there are contact addresses, and even warnings for the copyright protection. Information comprises factual data; data is represented in the way that it can be authenticated. It is not clear who the sponsor of the web site is, who owns it or who the author is. The real author of the web site is unknown and there is no contact address. Objectivity Information is complete, intelligible, timely, accurate and unprejudiced. Currency Data about the information source, time and place of its origin are up to date and complete. Coverage Communication functions; there are contact addresses and links to other sites or web pages. Information does not Information does not contain particular contain actual, evidence about the approved facts; Represented data; information data cannot be comprises false and checked. inaccurate data. Information is not Information is not objective; it contains factual; it strives to author's personal mislead the user opinions and regarding the content therefore is of the information. prejudicial. Data about the Data about the information source, information source, time and place of its time and place of its origin are origin do not exist or incomplete or outthey are not updated. of-date. Communication Communication does functions; there are not function; future contact addresses Interaction with the and links to other website is not clear. sites (web pages), 17 but they are sometimes broken. Table 2: Relationship between the 5 criteria for the evaluation of information on the internet (Tudjman and Mikelic, 2003) While the above criteria may be used to evaluate websites as well as the information found on these sites, there must still be a method used behind filtering out the misinformation and disinformation (Keshavarz, 2014). Again, numerous researchers have proposed their own models in this regard. For example, a ‘Misinformation Prevention Framework’ has been developed by Koohang and Weiss (2003) which users on the internet may use when they are seeking information. This model may be used when users find themselves in any one of the three stages; these being: Information identification, information analysis and information organisation (Keshavarz, 2014). Several considerations must be kept in mind depending on the stage the users are situated. Koohang’s and Weiss’ (2003) model is as follows: Misinformation Prevention Framework (Koohang and Weiss, 2003) The Misinformation Prevention Framework consists of three phases which are: information identification, information analysis, and information organisation. For organisational purposes, this framework relies heavily on the interdisciplinary and cross-functional processes of the organisation (Koohang and Weiss, 2003). Once each of the three phases has been completed, a checkpoint is reached where the information architect receives assistance from personnel in the interdisciplinary and cross-functional processes in the organisation. The purpose of each checkpoint is to ensure that all the required steps were following correctly in each phase (Koohang and Weiss, 2003). If the evaluation within the checkpoint is successfully accomplished then the information architect may move onto the next phase; if not, then that phase requires extra examination. The three phases are as follows: 18 Phase I Information Identification No Check Yes Phase II Information Analysis No Check Yes Phase III Information Organisation No Check Yes Disseminate Figure 1: Misinformation Prevention Framework (Koohang and Weiss, 2003) Phase 1 – Information Identification Phase 1 involves the identification of information. This information must be defined and described. Furthermore, the nature of the information must also be identified. In addition to this, the information needs must also be described which may be carried out using various techniques such as direct observation and surveys (Koohang and Weiss, 2003). Due to the fact 19 that this phase deals with information identification, it is important to be aware of the five different types of information sources. These are:      Advocacy – sponsorships by organisations Marketing/business – business enterprises bidding to advertise or sell their products News – news entities providing news stories Informational – factual information sponsored by the government or educational institutions Personal – entities distributing personal information Identifying information sources during this phase will help when dealing with biased information in the next phase (Koohang and Weiss, 2003). Phase 2 – Information Analysis This phase is the most detailed and crucial phase because its purpose is to outline the complete functionality of the information, as well as its source, architecture and content (Koohang and Weiss, 2003). This outline will then work as the groundwork for which decision making will be based on. This phase analyses criteria such as accuracy, relevancy, and completeness of the information. It also evaluates the authorship of the resource and checks to see that the information resource is certified (Koohang and Weiss, 2003). If the information sources intend to be truly informative then they should not be biased and must contain factual information. If the author intends to put forward their own opinions, then this should be stated clearly and not be presented together with the facts. The date that the information was modified should also be indicated because information may change over time and out-dated information may sometimes be misinformative (Koohang and Weiss, 2003). If the resource is sponsored by an external organisation, then the possible motives of said organisation must be taken into consideration to account for any possibility of bias. The author should provide their contact information such as their email address so that they can be contacted for clarification purposes. Lastly, any information inconsistencies based on the viewer’s knowledge in addition to inconsistencies within the resource must be noted (Koohang and Weiss, 2003). 20 Phase 3 – Information Organisation On completion of the information analysis phase, the information must then be organised before being disseminated. In essence, this phase largely deals with the semantics and presentation of the collected information (Koohang and Weiss, 2003). The information organisation phase involves carrying out the following tasks: customising the acquired information for a particular environment, presenting the information in an easy to understand manner, it must be clearly labelled, it must include an information map, key points must be emphasized, and it must possess visual clarity (Koohang and Weiss, 2003). Once again the information must be checked in order to ensure that the phase was carried out properly. Finally, the information must continuously be evaluated and made sure that it is always up-to-date in order to maintain its credibility (Koohang and Weiss, 2003). 2.7 Conclusion The internet’s position as the world’s largest information sharing platform has proven to be extremely beneficial and well as detrimental to its users. The internet’s lack of governance and authority, and the rise of social media has made it relatively easy to deceive other users (Fallis, 2009). The exchange of information on the internet has become an interactive and social relation and users often have opposing interests. This rise in information sharing has brought with it an increased rise in the amount of misinformation and disinformation present on the internet due to users’ contrasting views and interests (Tudjman and Mikelic, 2003). With the move of information resources from libraries and physical hard copies to the worldwide internet platform, traditional methods of information evaluation are no longer as effective as they were in ensuring the credibility of information (Keshavarz, 2014). It is therefore becoming increasingly important to develop global methods for information evaluation on the internet in order to decrease the amount of misinformation and disinformation on the internet and increase the awareness of users to misleading information (Keshavarz, 2014). This research paper will ultimately aim to create a taxonomy that may be used to help internet users identify sources of misinformation and more specifically disinformation on the internet. This will be done by reviewing current models and taxonomies that have been proposed by other authors in the fields of misinformation and disinformation evaluation and producing a 21 unique taxonomy that internet users may be able to use to evaluate sources of ‘information’ on the internet. 22 CHAPTER 3 Current Frameworks 3.1 Introduction When it comes to the evaluation of information on the internet and assessing whether this information may in fact be misinformation or disinformation, the most prominent method of evaluation used by authors worldwide is through assessing the information’s credibility. It may be said that there is a direct relationship between information quality and information credibility (Keshavarz, 2014). That is, information sources that are of low credibility will result in low information quality and vice versa. The term credibility implies truth which means that credible information can be relied on. Information sources that may lack truth, such as misinformation and disinformation, will also lack credibility therefore indicating that this ‘information’ cannot be fully relied on (Gackowski, 2006). Disinformation, due to its intention to deceive its audience, may be seen to be more dangerous than misinformation as well as less credible because disinformation will tend to go against more of the known properties of credible information. In this chapter I will outline what makes information credible and will describe the various properties and characteristics of credible information. This information will then be used to build my proposed taxonomy in the next chapter which will be a combination of credibility characteristics as well as a misinformation prevention framework. 3.2 Discussion Misinformation and disinformation on the internet are largely caused by three distinct problems. These are: the nature of internet structure, the architecture of the internet, and the traditional causes for misinformation/disinformation (Keshavarz, 2014). The architecture of the internet is associated with problems such as are the lack of central authority and data malleability. Traditional causes for misinformation are problems such as human error, misconduct, a lack of currency, bias and the removal of information from context (Keshavarz, 2014). When it comes to disinformation, the causes differ because they are associated with the source’s intent to falsify the information which may be political, instructional or malicious entertainment (Keshavarz, 2014). Ensuring that information accessed on the internet is of a high quality, and is therefore credible, plays a big part in reducing the amount of misinformation and disinformation found on the internet. 23 3.2.1 Credibility Although credibility has been a key consideration in numerous areas of research and practice, it has not been given significant attention regarding information on the internet despite its high importance (Liu, 2004). The research that has been carried out in this field has found that credibility is most widely investigated at the website/structural level (Keshavarz, 2014). Numerous authors who have studied the evaluation of material on the internet state that when it comes to the evaluation of this material, one should assess this online material by assessing its credibility which is an indicator of the information’s quality (Keshavarz, 2014). Information, misinformation and disinformation carry different levels of credibility depending on the number of credibility criteria they go against. For example, websites that provide users with information will possess many, if not all of the criteria that are characteristic of credible information; whereas, websites that attempt to disinform its viewers will only possess a very small number (if any) of credibility criteria. Before describing the various credibility criteria, it must be made clear that it is assumed that internet users possess two basic skills which are critical thinking and information literacy. These two terms are sometimes used interchangeably as they are very similar (Keshavarz, 2014). Both these skills are required to effectively evaluate and use information found on the internet (Weiler, 2004). There are no universally agreed definitions of the two terms however; the term critical thinking has been discussed in many fields of study and is regarded as a key skill used in evaluation and is considered a fundamental competency needed by internet users (Gilster, 1997). Information literacy has been described as a “socio-technical practice, incorporating knowledge of the epistemological aspects of the information sources as well as of the technology and systems that make up their material dimension” (Sundin and Francke, 2009). Training users of the internet in the skills of critical thinking and information literacy are two issues that may increase web literacy, and which may be seen as a method to tackle and help said users identify sources of misinformation and disinformation. Metzger (2007) added that in addition to these two competencies, users of the internet must also be willing to evaluate the online material they come across otherwise the credibility of these sources will be of least importance when users are not motivated to carefully evaluate and examine the content of the information. 24 3.2.2 Credibility Criteria The assessment of information credibility is closely associated to content evaluation, where there is a large library of literature that considers the credibility of content as the chief indicator of information quality (Robins and Holmes, 2008). There are numerous criteria that have been put forward regarding web content evaluation; however, the five most prominent criteria are authority, accuracy, objectivity, currency and scope/coverage (Tudjman and Mikelic, 2003; Keshavarz, 2014; Tate, 1999; Beck, 1997; Smith, 1997; Meola, 2004). Table 2 in the literature review section of this paper describes the above five mentioned criteria and how they are evident in information and lacking/non-existent in misinformation and disinformation. Their definitions are as follows: Authority - may be assessed by noting who authored the site and whether contact information is provided for that person or organization, what the author’s credentials, qualifications, and affiliations are, and whether the website is recommended by a trusted source. Accuracy - the degree to which a website is free from errors, whether the information can be verified offline, and the reliability of the information on the site. Objectivity - involves identifying the purpose of the site and whether the information provided is fact or opinion, which also includes understanding whether there might be commercial intent or a conflict of interest on the part of the source, as well as the nature of relationships between linked information sources. Currency - refers to whether the information is up to date. Scope/Coverage - the comprehensiveness or depth of the information provided on the site. Table 3: Most Prominent Credibility Criteria (Tudjman and Mikelic, 2003) There are numerous other criteria that have been identified by other researchers. Such criteria include the likes of accessibility, relevance, durability, privacy, variety of sources, workability, and presentation credibility/website design (Keshavarz, 2014). Most of these criteria are often 25 evaluated through checklists however, users who are not sure about what a lot of these criteria entail will find them hard to apply to the checklist method (Keshavarz, 2014). 3.2.3 Misinformation Prevention Framework The Misinformation Prevention Framework was presented my Koohang and Weiss in 2003. It is an evaluation model that users may apply in the process of information seeking and evaluation. The model consists of three stages in which an internet user may find themselves situated in a point in time. The three stages are: information identification, information analysis, and information organization (Keshavarz, 2014). Once each of the three phases has been completed, a checkpoint is reached where the information architect receives assistance from personnel in the interdisciplinary and cross-functional processes in the organisation. The purpose of each checkpoint is to ensure that all the required steps were followed correctly in each phase (Koohang and Weiss, 2003). If the evaluation within the checkpoint is successfully accomplished then the information architect may move onto the next phase; if not, then that phase requires extra examination. Phase 1 involves the identification of information. This information must be defined and described. (Koohang and Weiss, 2003). Due to the fact that this phase deals with information identification, it is important to be aware of the five different types of information sources. These are:      Advocacy – sponsorships by organisations Marketing/business – business enterprises bidding to advertise or sell their products News – news entities providing news stories Informational – factual information sponsored by the government or educational institutions Personal – entities distributing personal information Identifying information sources during this phase will help when dealing with biased information in the next phase (Koohang and Weiss, 2003). Phase 2 is the most detailed and crucial phase because its purpose is to outline the complete functionality of the information, as well as its source, architecture and content (Koohang and Weiss, 2003). This outline will then work as the groundwork for which decision making will be based on. This phase analyses criteria such as accuracy, relevancy, and completeness of the 26 information. It also evaluates the authorship of the resource and checks to see that the information resource is certified (Koohang and Weiss, 2003). If the information sources intend to be truly informative then they should not be biased and must contain factual information. If the author intends to put forward their own opinions, then this should be stated clearly and not be presented together with the facts. The date that the information was modified should also be indicated because information may change over time and out-dated information may sometimes be misinformative (Koohang and Weiss, 2003). If the resource is sponsored by an external organisation, then the possible motives of said organisation must be taken into consideration to account for any possibility of bias. The author should provide their contact information such as their email address so that they can be contacted for clarification purposes (Koohang and Weiss, 2003). Phase 3 is known as Information Organisation and is where the information must then be organised before being disseminated. In essence, this phase largely deals with the semantics and presentation of the collected information (Koohang and Weiss, 2003). 3.3 Conclusion Information credibility and the Misinformation Prevention Framework are the two core frameworks that I will carry forward into the next chapter of this paper: the proposed framework chapter. The framework that I will propose is fundamentally a taxonomy as it will simply be a classification of material into three categories which are information, misinformation and disinformation. The below diagram (Figure 2) is essentially what I aim to incorporate into my taxonomy along with a small number of enhancements such as a numerical scale to gauge the severity of misinformation/disinformation. It will help the user establish the nature of the material they are viewing by placing them in either the information, misinformation or disinformation category. 27 Figure 2: Proposed Framework 28 CHAPTER 4 Proposed Framework 4.1 Introduction This chapter comprises a proposed framework, or more specifically a taxonomy, that may be used to ascertain the level (if any) of misinformation or disinformation present on any particular webpage. Provided the taxonomy works successfully, it will be able to firstly identify/confirm whether a webpage is highly credible and provides information to its viewers or whether it is of low credibility and therefore may contain misinformation or disinformation. Secondly, the taxonomy will provide a guide/scale to distinguish between misinformative and disinformative sites based on the level of credibility of the material provided. I will be using Koohang and Weiss’ Misinformation Prevention Framework as a base for my framework. I will then add to it several different credibility criteria in addition to my own scoring/rating system. The final product is a taxonomy that is able to classify and place material into three categories which are information, misinformation and disinformation. 4.2 Details of Proposed Framework What my research aims to produce is a score for each webpage based on its structure and content. Webpages that carry a low score will most likely be informative whereas webpages that carry a relatively high score will either be misinformative or disinformative, depending on the exact score achieved. A middle rated score is an indication of a site that provides misinformation, whereas a high score is an indication of a site that provides disinformation. The first Phase in Koohang and Weiss’ Misinformation Prevention Framework (figure below) is information identification. This implies describing and defining the information as well as identifying the nature of this ‘information’. When identifying information, it is important to note the five different types of information sources as these sources may tend to alter the information they put forward in order to influence the behaviour and thought processes of their audience. This process of identifying information sources is important as it helps deal with biased information going forward (Koohang and Weiss, 2003). The five different types of information sources are as follows: 29      Advocacy – sponsorships by organisations or government Marketing/business – business enterprises bidding to advertise or sell their products News – news entities providing news stories Informational – factual information sponsored by educational institutions Personal – entities distributing personal information eg) blog or chat forum Phase I Information Identification No Check Type of Information Source Yes Phase II Information Analysis No Check and Evaluate Credibility Criteria Yes Phase III Information Organisation No Removed from Framework Check Yes Disseminate Figure 3: Misinformation Prevention Framework Adaptation 30 From the five different types of information sources described above, their scores will be given as follows: Information Source Score Informational 0 News 2 Advocacy 3 Personal 4 Marketing/Business 5 Table 4: Information Sources and Related Scores This will be the starting point of each webpage’s score and will most likely set the tone for the rest of the scoring process. Phase II of the Misinformation Prevention Framework is known as the Information Analysis Phase and it involves the majority of the scoring process entailed in the proposed framework. It is the most crucial phase because its purpose is to provide the groundwork for which decision making will be based on by outlining the functionality, architecture and content of the information source (Koohang and Weiss, 2003). This Phase involves assessing the information credibility of the specified webpage by analysing criteria such as authority, accuracy, objectivity, currency and scope/coverage. These criteria were described in detail in the previous chapter, therefore I will simply present my proposed scoring system relating to them in the table below. In addition to the credibility criteria already listed, I have added a couple more to the list based on my research findings. These additional criteria have proven to be important criteria that have been put forward by various authors in papers aimed at evaluating the credibility of online information sources (Metzger, 2007). The criteria I have added are presentation, variety of sources and advertising. I have added these criteria because research carried out in an online experiment of a similar nature found that site design/presentation elements were the most frequently used criterion when evaluating the credibility of a webpage. Another finding of this research was that the information source’s motives are a key factor in determining the source’s intent. For example, the presence of advertising/clickbait links and their selling intent on their sites (Metzger, 2007). 31 Information Misinformation Disinformation 0 1 2 Authors and sponsors are known; furthermore, there are contact addresses, and even warnings for the copyright protection. It is not clear who the sponsor of the web site is, who owns it or who the author is. The real author of the web site is unknown and there is no contact address. Accuracy Information comprises factual data; data is represented in the way that it can be authenticated. Information does not contain particular evidence about the Represented data; data cannot be checked. Information does not contain actual, approved facts; information comprises false and inaccurate data. Objectivity Information is complete, intelligible, timely, accurate and unprejudiced. Information is not objective; it contains author's personal opinions and therefore is prejudicial. Information is not factual; it strives to mislead the user regarding the content of the information. Currency Data about the information source, time and place of its origin and update are complete. Data about the information source, time and place of its origin are incomplete or out-of-date. Data about the information source, time and place of its origin do not exist or they are not updated. Coverage Communication functions; there are contact addresses and links to other sites or web pages. Communication functions; there are contact addresses and links to other sites (web pages), but they are sometimes broken. Communication does not function; future Interaction with the website is not clear. Presentation Professional, attractive, and consistent page design, including graphics, logos, colour schemes, etc. Several, but not all characteristics of professional webpage design and presentation present. Lack of professional touch, unattractive, inconsistent page design. Variety of Sources Large variety of sources share same or similar material/thoughts. Eg) >10 Relatively small number of sources share same or similar material. Eg) 5<X<10 Small variety (if any) of sources share same or similar material. Eg) <5 SCORE/ RATING Authority 32 Advertising No advertising or sponsorships present on webpage or material provided. Sponsorship of webpage by a particular entity or organisation. Sponsorships and/or advertisements present on webpage as well as pop-up windows and clickbait links. Table 5: Relationship between credibility criteria for the evaluation of information on the internet with corresponding scores Phase III of the Misinformation Prevention Framework is Information Organisation which is where the information is analysed and organised for dissemination (Koohang and Weiss, 2003). This phase involves information customisation for a specific environment as it is prepared for dissemination; therefore, for the purpose of our scoring system we will overlook this phase as it is not directly related to our end goal of evaluating the information we are exposed to online. Our evaluation system will therefore only consist of two distinct phases. The resultant framework is the one presented below (Figure 4) which has been named the ‘Proposed Information Evaluation Framework’: 33 Phase I Information Identification No Check Type of Information Source Yes Phase II Information Analysis No Check and Evaluate Credibility Criteria Yes Add Scores from Phase I and II Yes Determine nature of information source Figure 4: Proposed Information Evaluation Framework 4.3 How it Works Step 1: The first step a user wishing to evaluate an online resource is to identify the type of information source that they are looking at as set out in table 4. The type of information source identified will be the base/starting score before moving onto the next step. This step provides an indication of how the majority of the following scoring will most likely go. For example, if the type of information source is informational or news, one may expect the evaluation of the criteria to lean towards informative information; whereas, if the type of information source is identified to be a personal blog or a site ridden with advertisements, then one should expect the evaluation of credibility criteria to lead towards misinformation and even disinformation. Step 2: The user then goes through the credibility criteria listed in table 5 and gives a score against each criteria whist referring to the online resource they are evaluating. 34 Step 3: The user adds up the scores from step 1 and all the credibility criteria in step 2, attaining a total score that will be used to assess the resource’s credibility. Step 4: The final step is to compare the final score attained with the table below (table 6) which illustrates what type of information source one might be looking at with regard to the total score attained. It is important to note that although a high score indicates that the source could be misinformative or disinformative, it is not definitive but rather an indication that there is a high chance that the source may not be truthful and informative as it does not meet the criteria of an academic resource. This means that a high score does not rule out that a source is not informative. Total Score Nature of Information Source Between 0 and 5 Information Between 6 and 12 Misinformation More than 12 Disinformation Table 6: Total Score and Probable Type of Information Source 4.4 Conclusion This chapter outlined the construction as well as the steps involved in using my proposed framework. The next stage and following chapter of this paper is to ‘test’ the above framework using sites found online and evaluate the credibility of the ‘information’ they provide. Ideally, I aim to provide readers of this paper with case studies of evaluations I have carried out with the intention of evaluating both an informative site as well as a disinformative site for reference purposes. 35 CHAPTER 5 Discussion 5.1 Introduction This chapter will serve as the practical aspect of this paper as it will be using real world examples to observe whether the proposed Information Evaluation Framework functions as it is designed to. I will attempt to incorporate both sides of the information/disinformation spectrum so that readers observe the differences between the two types of sources. This will be followed by a summary on whether the results from the case studies are as expected which will determine the future work regarding this framework. 5.2 Interpretation of Proposed Framework 5.2.1 Case Study 1 The first webpage I will be evaluating is an academic resource from a web platform (academia.edu) that allows academics to share papers they have written. It is a short paper written regarding global warming and development and is titled ‘The Unfair Narrative on Global Warming and Development: Why it must be challenged’. The exact paper used can be found here: https://www.academia.edu/27148803/The_Unfair_Narrative_on_Global_Warming_and_Dev elopment_Why_it_must_be_challenged . Below is a copy of my scoresheet for the mentioned article/webpage which is followed by reasoning behind each criteria evaluation. Please see scoresheet (Figure 5) on next page labelled Scoresheet - Case Study 1. 36 Figure 5: Scoresheet – Case Study 1 37 The results of the evaluation were as desired. The first step was to identify the type of information source which in this case was ‘Information’ which carries a score of 0. The reason why this article was classified as this type is because it is presented as factual information through an academic resource. The next step was to go through each of the evaluation criterion: ‘Authority’ received a score of 0 because the author was known and their contact details were present to allow the user to contact the author if need be. I presented ‘Accuracy’ a score of 1 because although the information provided does seem factual, there was no evidence about the represented data present. ‘Objectivity’ was scored 0 due to the fact that the material was complete, intelligible, and timely and absent of the author’s personal opinions. The ‘Currency’ criterion was given a score of 1 because although the date of the article is present, it lacks a place of origin as well as other data about the information source. “Coverage’ was scored a 0 due to the communication functions present on the article. The reader has access to communicate with the author through various channels such as a direct message through the Academia.edu portal, email address, Facebook and Linkedin links. The author also provides links to articles supporting the facts he has put forward. I gave ‘Presentation’ a score of 0 because the resource was presented professionally with consistent page design. Although the page has minimalistic features, one can tell that the page is modern and up to date with current professional web application design. The criterion ‘Variety of Sources’ was also scored a 0 because a simple Google search for similar material (Global warming subsidising) provides the user with multiple sources (>10) relating to similar topics. Lastly, due to its academic nature, ‘Advertising’ was also scored a 0 because there was no advertising or sponsorship present on the page. Step 3 of the process was to add the scores from steps 1 and 2 together which in this example gave us a final total of 2. This means that looking at the table in step 4, the evaluation suggests that the article possesses the criteria of a credible resource and it may be trusted. 38 5.2.2 Case Study 2 The second evaluation I carried out was on a website called ‘RealFarmacy’ which claims to disseminate ‘healthy news and information’. The specific article I evaluated was an article concerning a particular brand of weed killer that when used may have fatal consequences for its users. The specific article evaluated can be found here: http://www.realfarmacy.com/dangerous-roundup/. Below is a copy of my scoresheet for the mentioned article/webpage which is followed by reasoning behind each criteria evaluation. Please see scoresheet (Figure 6) on next page labelled Scoresheet - Case Study 2. 39 Figure 6: Scoresheet – Case Study 2 40 The results of the evaluation in case study 2 were satisfactory. Beginning at the first step of identifying the type of information source led me to providing the article a score of 2 which is representative of ‘News’. The reasoning behind this classification is because this article shared characteristics that are present in news articles. For example, the article repeatedly refers to previous studies that have been carried out, however it fails to provide the user with links to these studies or even provide references throughout or at the end of the article. I viewed this characteristic in a similar esteem as conventional news as I have personally come across this behaviour by conventional news sources. Another reason for classifying this article as news is because it simply cannot fall into any of the other categories listed. Its closest affiliation may be to ‘Personal’ however the author is not presenting her own opinion but rather citing information from different sources. My evaluation of the credibility criteria was as follows: ‘Authority’ was given a score of 1 because, although an author name is given, there is no additional info provided. ‘Accuracy’ was scored a 1 due to the fact that data presented in the article was not referenced nor were links to the data provided. On the other hand, the information provided did appear complete and intelligible which led me to provide ‘Objectivity’ with a score of 0. The ‘Currency’ criterion however, was given a score of 2 because data about the information source, time and place of origin do not exist. ‘Coverage’ was also given the same rating (2) because a reader is not able to communicate with either the author of the article or the website owner. There are no standard communication features present on the site. That being said, there are links to social media platforms in the page footer, however they do not function as clicking on them simply reopens the article in another tab. The ‘Presentation’ of the site was scored a 1 because the site possessed some, but not all the characteristics of a professional website design. ‘Variety of Sources’ was scored a 0 since a Google search of the title of the article provides numerous pages of articles based on similar subject matter. Lastly, the ‘Advertising’ criterion was scored 1 due to the fact that numerous other articles are advertised on the webpage. This is evident by a panel of articles on the right side of the page as well as a couple at the bottom of the page, in addition to a tag that pops up on the bottom right hand side of the page once the user scrolls to the bottom of the article. Adding the scores from the previous two steps as described in step 3 provides us with a total score of 10 which is an indication that the webpage may be a source of misinformation based on the structure of the webpage and the writing style of the author. While I do not perceive the claims and material found on this website to be completely false and untrustworthy, my 41 evaluation simply points out that the page lacks the criteria academic sources have that make them credible. The material on this site may be true in most cases, however, it is difficult to know which articles are backed academically and which ones are not. My evaluation therefore may rule the webpage to be misinformative but this is only on the grounds that its structure lacks several characteristics of a credible resource. 5.3 Conclusion This chapter’s aim was to put the proposed framework into practice to assess whether it functions and behaves as specified. From the results of the two examples carried out, I do believe that the framework is on the right track, however I have made observations regarding the limitations of the framework which will be discussed in the following chapter. Whilst there are similar methods of evaluation presented by other authors, I am yet to come across one that assigns a score to a particular resource and makes a judgement based on that. This can be seen as both an advantage and a disadvantage as the framework simply provides an indication of the nature of the information, regardless of whether it is actually truthful or not. 42 CHAPTER 6 Conclusion and Future Work 6.1 Introduction The previous chapter was used to discuss the proposed framework through the use of real world examples (two case studies). This chapter serves as a summary and final conclusion of this dissertation, illustrating how well the research questions were answered and will also provide recommendations for future work. 6.2 Final Summary and Conclusion This research’s aim was to provide answers to the research questions of:   How does one distinguish between information, misinformation and disinformation?  internet? What frameworks are available to identify misinformation and disinformation on the How is the credibility of information on the internet evaluated? I believe this was done satisfactorily through the following means: the research was able to describe the three concepts of information, misinformation and disinformation. Furthermore, the research distinguished the three concepts from each other, outlining both similarities and differences which will provides readers with an understanding of the concepts before moving onto the proposed framework and taxonomy. Describing and distinguishing the three concepts will also enable internet users to be more aware towards what and how misinformation and disinformation present themselves. This paper was able to point out and describe frameworks that are currently available to identify misinformation and disinformation on the internet. It must be noted that the number of frameworks covered was not exhaustive, however most existing frameworks regarding similar purposes do base themselves on a similar concept of evaluating through assessing credibility. This brings us to the last research question of how is information credibility evaluated on the internet? This is predominantly done through the checking of certain criteria that are present on most if not all trusted academic sources. The list of possible criteria one could use is extensive and would but impractical to use all of them. It is also important to note that some 43 criteria are more effective than others when it comes to evaluating credibility. This research paper selected the most widely used and prominent criteria previously used by other authors. These criteria were then incorporated into a framework that has been used to help prevent the dissemination of misinformation, that is, Koohang and Weiss’ Misinformation Prevention Framework (2003). Through incorporating the most prominent criteria with a modified version of the Misinformation Prevention Framework, I was able to create an original taxonomy/framework that users of the internet may use to evaluate the material they come across online. The created taxonomy assigns a score to a particular resource and makes a judgement on the credibility of the resource based on that score attained. The score given to a particular resource places it in one of the categories of information, misinformation or disinformation. A highly important point to note is that the score given to a particular resource does not explicitly define it. The score given is simply an indication to the possible nature of the resource. It is a comparison between the resource being evaluated and the characteristics of a normally trustworthy academic internet resource. A score pointing a resource towards misinformation or disinformation implies that the resource lacks the structure and characteristics of a credible web resource and that it might not be fully informative. It does not definitively imply that the resource is not informative and cannot be trusted; it is simply a guide. 6.3 Future Work Due to the fact that the created taxonomy was only recently conceived, I believe that it will be able to be improved in future by other authors with different knowledge and perspectives. In particular, I believe that the scoring system may be enhanced once the taxonomy has been extensively used and scrutinised. Reason being, the scoring system currently used was not put through comprehensive testing. In addition, the current system is not definitive but rather an indication as to the nature of the resource. For example, a high score is indicative of a disinformative source, however it does not necessarily guarantee that it is disinformation. In future, it would be beneficial to create a new taxonomy of update the current one to incorporate this and produce more conclusive and accurate judgements. 44 References Beck, S., 1997. The good, the bad & the ugly: Or, why it's a good idea to evaluate web sources. Retrieved May, 14, pp.2003. Chamberlain, P.R., 2009. Twitter as a Vector for Disinformation. School of Computer & Security Science, Edith Cowan University, Australia. Fallis, D., 2009. A conceptual analysis of disinformation. Fallis, D., 2014. A Functional Analysis of Disinformation. iConference 2014 Proceedings. Fallis, D. and Carlin, G., 2011. Floridi on disinformation. Etica & Politica, 13 (2), pp.201-214. Fetzer, J.H., 2004. Disinformation: The use of false information. Minds and Machines, 14(2), pp.231-240. Fisher, J., Burstein, F., Lynch, K. and Lazarenko, K., 2008. “Usability usefulness= trust”: an exploratory study of Australian health web sites. Internet Research, 18 (5), pp.477-498. Floridi, L., 1996. Brave. Net. World: the Internet as a disinformation superhighway? The Electronic Library, 14 (6), pp.509-514. Fox, C. J., 1983. Information and misinformation: An investigation of the notions of information, misinformation, informing, and misinforming., Westport, CT: Greenwood. Gackowski, Z.J., 2006. Quality of informing: Bias and disinformation philosophical background and roots. Issues in Informing Science and Information Technology, 3. Gilster, P. and Glister, P., 1997. Digital literacy. Wiley Computer Pub. Hernon, P., 1995. Disinformation and misinformation through the Internet: Findings of an exploratory study. Government Information Quarterly, 12(2), pp.133-139. Karlova, N.A. and Fisher, K.E., 2012. Plz RT”: A social diffusion model of misinformation and disinformation for understanding human information behaviour. Proceedings of the ISIC2012 (Tokyo). Lee, AYL & So, CYK (2014). Alfabetización mediática y 45 alfabetización informacional: simi-litudes y diferencias. Comunicar, 21(42), pp.137146. Karlova, N.A. and Lee, J.H., 2011. Notes from the underground city of disinformation: A conceptual investigation. Proceedings of the American Society for Information Science and Technology, 48(1), pp.1-9. Keshavarz, H., 2014. How Credible is Information on the Web: Reflections on Misinformation and Disinformation. Infopreneurship Journal, 1 (2), pp.1-17. Koohang, A. and Weiss, E., 2003. Misinformation: toward creating a prevention framework. Information Science. Kumar, K.K. and Geethakumari, G., 2014. Detecting misinformation in online social networks using cognitive psychology. Human-centric Computing and Information Sciences, 4 (1), pp.1-22. Liu, Z., 2004. Perceptions of credibility of scholarly information on the web. Information Processing & Management, 40 (6), pp.1027-1038. Meola, M., 2004. Chucking the checklist: A contextual approach to teaching undergraduates Web-site evaluation. portal: Libraries and the Academy, 4 (3), pp.331-344. Metzger, M.J., 2007. Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology,58(13). Oosterwoud, R., 2015. Disinformation in 21st Century Russia. Robins, D. and Holmes, J., 2008. Aesthetics and credibility in web site design. Information Processing & Management, 44 (1). Smith, A.G., 1997. Testing the surf: criteria for evaluating Internet information resources. Public Access-Computer Systems Review, 8 (3). 46 Stahl, B.C., 2006. On the difference or equality of information, misinformation, and disinformation: A critical research perspective. Informing Science: International Journal of an Emerging Transdiscipline, 9, pp.83-96. Stair, R. and Reynolds, G., 2013. Principles of information systems. Cengage Learning. Stvilia, B., Gasser, L., Twidale, M.B. and Smith, L.C., 2007. A framework for information quality assessment. Journal of the American Society for Information Science and Technology, 58 (12), pp.1720-1733. Sundin, O. and Francke, H., 2009. In search of credibility: Pupils' information practices in learning environments. Information research, 14(4). Tate, M.A. and Alexander, J.E., 1999. Web wisdom: How to evaluate and create information quality on the Web. CRC Press. Tudjman, M. and Mikelic, N., 2003. Information science: Science about information, misinformation and disinformation. Proceedings of Informing Science Information Technology Education, pp.1513-1527. Weiler, A., 2005. Information-seeking behavior in Generation Y students: Motivation, critical thinking, and learning theory. The Journal of Academic Librarianship, 31 (1), pp.4653. 47 Appendices Figure 7: Blank Scoresheet 48