EJISDC (2012) 50, 5, 1-14 1

ICT EVALUATION: ARE WE ASKING THE RIGHT QUESTIONS?

Ricardo Gomez Shaun Pather
Information School Informatics and Design
University of Washington Cape Peninsula University of Technology
Seattle Cape Town
USA South Africa
[email protected] [email protected]

ABSTRACT
The evaluation of information and communication technologies (ICT) in development
activities has metamorphised through different phases in the last 30 years. In this paper we
explore the experiences of ICT evaluation in the broad business environment and draw
parallels with the ICT for development (ICTD) environment. The paper motivates the need
for a fundamental paradigm shift in ICTD evaluation. We argue that it is not sufficient to
focus on the easily measurable tangible and quantifiable benefits of ICT. We motivate that
the intangible benefits of ICT on development such as empowerment, self-esteem and social
cohesion are more important from a developmental perspective. Consequently, a more
detailed exploration of the theoretical and methodological implications of this shift in the
ICTD field is warranted.

Keywords: Evaluation, ICT, development, impact, intangible

1. INTRODUCTION
The evaluation of the effectiveness of Information and Communication Technologies (ICT)
has been evolving since the first commercial deployment of computers. There has been much
debate in the field of evaluation with regards to approaches to assess the benefits and impact
of ICT, which are elusive and frequently intangible. The introduction of ICT into
international development activities since the late 1990s added a new layer of challenges to
the already elusive subject of ICT evaluation. This paper discusses the following question:
how can we assess the direct and indirect benefits of the use of information and
communication technologies in the context of international and community development
efforts?
The application of ICTs as a developmental tool has progressed through various
phases over time. Thirty years ago the use of ICT for development (ICTD) was viewed by
skeptics as a needless luxury: “How can we prioritize computers when people need water and
roads?” was a typical response by critics in the late 1980s. Twenty years ago ICTs became a
panacea that was euphorically believed to solve every development problem: “ICTs will
bring about a new world”, is another example of a typical euphoric claim as we approached
the end of the millennium. Ten years ago, however, much cynicism followed the euphoria,
fueled in part by the many anecdotal stories of failure and success of ICT experiences in
development that characterized much of the ICTD environment during the first few years of
the new millennium. Despite widespread investment in ICT by governments and donor
agencies, questions still remain as to what are the impacts, if any, of ICT in development.
Today, a handful of researchers have started to ask hard questions about the
evaluation of ICT impacts, wondering whether they do exist, and whether there is a way to
reliably measure them (e.g., Heeks, 2009a, and TASCHA, 2010). In recent years,
nonetheless, ICT and development initiatives are being either abandoned, mainstreamed into
broader sector development themes (environment, agriculture, gender, education, etc.), or
“gadgetized”: ICTD is becoming a lab where numerous technology applications, devices and

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 2

gadgets are developed, hoping they will be adopted and used by people working in
international development contexts.
In reflecting on the ICTD evaluation landscape, we suggest the ICTD evaluation field
may be too narrowly focused on measuring the tangible and quantifiable economic benefits of
ICT for development, and as a result we may have been “barking up the wrong tree”. Have
we been so concerned with evaluating the tangible social and economic impacts that we have
neglected other intangible impacts, which may be equally or more important for human
development than the tangible and quantifiable ones? We argue that in the early days of
ICTD we were focused on evaluating the outputs (counting number of computers, number of
users, etc), and eventually turned our focus towards tangible outcomes related to economic
growth such as business opportunities, income generation, new markets, etc. This is
convergent with the modernization paradigm of development as transfer of technology for
economic growth. To elaborate this idea, we explore the experiences of ICT evaluation in the
business world, which has a longer history of ICT evaluation, and draw some parallels which
inform how the goalposts in ICTD evaluation could be shifted. We conclude this paper by
suggesting a fundamental shift in the way we think about ICTD evaluation. Finally, we point
the way towards a mindset which focuses on both tangible and intangible contributions of
ICT to human development.

2. THE PROMISES OF ICT

One of the most prominent early proponents of the benefits of ICT to human development
was then Vice-President Al Gore, who professed that " …we will derive robust and
sustainable economic progress, strong democracies, better solutions to global and local
environmental challenges, improved health care and ultimately, a greater sense of
stewardship of our small planet… [ICT] will help educate our children… it will be a
metaphor for democracy itself…” (Gore, 1994).
Two decades ago statements such as this gave both the developed and developing
worlds great hopes based on the potential of modern ICTs to assist an acceleration of
development efforts and impacts. This sort of rhetoric was in line with the classic view of
development as a ‘fast track’ to modernization, and of technology as an inevitable driver of
social change (‘technological determinism’). These views of development and technology
were coupled with an implicit threat to “get wired, or else”, based on the idea that “although
the costs of using ICTs to build national information infrastructures which can contribute to
innovative ‘knowledge societies’ are high, the costs of not doing so are likely to be much
higher” (Mansell & Wehn, 1998: 7). However, just as the technology world was jolted into
reality when the e-bubble burst in early 2000 (Remenyi, Grant & Pather, 2004), and the
business world was shaken by the financial meltdown of 2008, the development sector has
seen a slow melting of the dream of technology bringing about accelerated development,
wealth, and opportunity to the majority of the world’s poor.
With the dawn of the new millennium, and notably the two-phased World Summit on
Information Society (WSIS) shortly thereafter (Geneva, 2003 and Tunis, 2005) we have
witnessed a proliferation of research output in ICTD, supported by research agencies, non-
profit organizations, and academics. British researcher Richard Heeks (Heeks, 2009b)
suggests that hundreds of millions of US dollars are invested each year in ICTD projects, and
that the ICTD research area is growing significantly faster than other cognate areas.
Furthermore, he posits that the ICTD outputs to date reflect: (i) a bias to action and not a bias
to knowledge, (ii) a preference for what is narrowly descriptive, and (iii) a field that is not
analytical enough. Others suggest that shortcomings of this research area include a lack of
theory, conceptual definition, interdisciplinary approach, qualitative research, and
longitudinal research (van Dijk, 2006).

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 3

After 30 years, the field of evaluation of ICTD is still maturing. In the post hype era
of the millennium, there is an opportunity to refocus evaluation by moving beyond the
modernization paradigm and by closer introspection of intangible benefits. To inspire this
path, we will first discuss the experience of ICT evaluation in the business environment,
following which we present views regarding alternatives to the modernization approach to
development.

3. EVALUATING ICT - LESSONS FROM THE BUSINESS ENVIRONMENT1

The challenges of ICT evaluation are not entirely new. For many years, since the first
deployment of computers in business, researchers and practitioners have also grappled with
evaluation of its impact. Just as we currently recognize the intrinsic connection between ICT
and development, in the broad business sector there has long been the idea that the
implementation of ICT is indispensable to the provision of effective organizational services.
As a result, the implementation and management of ICT has presented both major
opportunities and challenges to businesses. Amongst these challenges, the increased
complexity of ICTs combined with the uncertainty and unpredictability associated with its
benefits and costs, pointed researchers to the development of sound evaluation methods that
offer companies a deeper insight into the impact of their ICT investment (Irani, Love & Zairi,
2000).
The business benefits of ICT gained much attention about twenty years ago, when
economist and Nobel Laureate Robert Solow characterized the computer age by saying that
“we see computers everywhere except in the productivity statistics” (Solow, 1987). This
anomaly became known as the productivity paradox of information technology, and there
were various reasons offered to explain this paradox, such as “deficiencies in [the]
measurement and methodological toolkit” and the “mismeasurement of outputs and inputs”
(Brynjolfsson, 1993:66). Other researchers before Solow had explored the potential business
benefits of ICT as early as forty years ago (e.g., Boyd & Carson, 1963; Gallagher, 1974;
Lucas, 1973). These early studies were concerned with whether technology was being
effectively used, and explained this effectiveness in a variety of ways ranging from relatively
simple accounting measures to complex multi-dimensional balanced score-card type metrics
(Bannister, Berghout, Griffiths & Remenyi, 2006); they also explored the use of surrogate
measures such as user satisfaction, service quality, individual, and organizational impact
(DeLone & McLean, 1992; Lomerson & Tuten, 2005; Seddon, Staples, Patnayakuni &
Bowtell,1999; Whyte, Bytheway, & Edwards, 1997). There are three phases that describe
how business interest in ICT evaluation has evolved over the years. These can be mapped
against three distinct eras of ICT deployment in business. Laudon and Laudon (2000:15-16)
describe the evolution of these eras: “In the 1950s the effects of IS [Information Systems, or
Information Technology] on organisations brought about merely technical changes, only
serving to automate clerical procedures. During the 1960s and 1970s IT had an impact on
managerial control, and from the 1980s onwards IS impacted upon core institutional activities
such as products, markets, suppliers and customers”.
A useful categorization of the phases of IT impact on organization was provided by
Zuboff (1988) who labeled these phases as Automate, Informate and Transformate: The
Automate phase focused on measurement of technical aspects of IT; the Informate phase
shifted towards evaluating the measurement of IT production or IT project management; the
Transformate phase focused on measurement of business benefits with a shift towards a

                                                            
1
We refer to the broad private sector business environment, as a means of reflection on the extant body of knowledge on IT
evaluation. This provides a useful basis to contrast with the application and evaluation thereof of ICTs for developmental
purposes in a broader societal context. 

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 4

service perspective. With the Transformate phase, ICT evaluation began to focus more on the
intangible aspects of business benefits, including issues such as trust, loyalty, and brand
improvement in evaluation frameworks.
Before we explore the possible parallels of these models in the field of ICTD
evaluation, let us examine in more detail, the intangible benefits of IT in the business world.

4. INTANGIBLE BENEFITS OF IT IN BUSINESS

In the business world, especially from the Transformate perspective described above, there
are various perceptions of the value that businesses derive from ICT. The value placed in ICT
is seen to become higher as its use in the organization progresses from being just a facility to
that of an enabler. Wiggers and colleagues suggested the IT Value Perception Model (Figure
1, below) to describe how IT value is perceived:
“The maturity of IT supply deals with the professionalism and quality of the IT
function in the organisation, with maturity being measured using a quality model such
as the Capability Maturity Model (CMM). Maturity of IT demand refers to the self-
awareness and self-consciousness of businesses to use and demand an appropriate
level of quality from their IT supportive organisations. The IT value perception
describes the perception of the executive management of the added value that IT
delivers to the company”. (Wiggers et.al., 2004:5)

High

IT is an enabler
Maturity of supply

IT is a partner

IT is a service

IT is a facility

Low

Low High
Maturity of demand

Figure 1: The IT Value Perception Model (Source: Wiggers et al., 2004: 4)

Similarly, in the ICTD environment as both maturity of demand and supply increase
with increasing investments in infrastructure, our evaluation foci should be tending towards
dimensions and metrics related to enablement. Thus, the context of ICT from a development
perspective must focus on ICT as an enabler, e.g., of socio-economic development. However
this is exactly where the evaluation challenge becomes more complex, as ICT may enable a
diverse set of outcomes that are difficult to link from an attribution or cause-effect
perspective.
The implementation of ICTs, regardless of the setting, can result in a variety of
benefits, some tangible, some intangible, and even some unexpected benefits (Kohli, Sherer,
& Baron, 2003). In the business domain, a tangible benefit is one which directly affects the
firm’s profitability, whereas an intangible benefit is one which can be seen to have a positive
effect on the firm’s business, but does not necessarily influence the firm’s profitability
directly (Remenyi, Money, Sherwood-Smith, & Irani, 2000). A number of approaches have

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 5

been adopted to evaluate both tangible and intangible benefits of IT. Examples of IT
evaluation techniques over the years have included system usage (Srinivasan, 1985; Trice &
Treacy, 1986); information value (Gallagher, 1974); user satisfaction (Bailey & Pearson,
1983; Shaw et al., 2002); and service quality (Jiang et al., 2002).
Hitt and Brynjolffson (1996) posit that the question of IT value is not a single one,
but rather consists of several related but distinct issues such as increased productivity, i.e., is
there now more output per given quantity of input? improved business profitability, i.e., has
the business been able to use IT to gain competitive advantage and earn higher profits than it
would have earned otherwise? improved value for consumers, i.e., what is the magnitude of
benefits that have been passed on to the consumers? In terms of the latter, the issue of profits
is not relevant in the ICTD domain. However improved value for both individuals and
communities is of increasing concern, and as suggested already, the measurement constructs
associated with such value is usually intangible.

IT Benefits in Tangible Intangible

Business
Quantifiable May be objectively Difficult to measure
(can be measured; objectively, e.g. obtaining
measured) e.g. increase in revenue; information faster;
reduction in costs improved customer
satisfaction
Unquantifiable Precise impact on Difficult to put a financial
(cannot be profitability cannot be value to the benefit, e.g.
measured or measured, increased customer
more difficult to e.g. better information; confidence; customers or
measure) improved security employees’’ perception of
the firms product.

Figure 2: Examples of Tangible and Intangible IT benefits in a business context

(Adapted from: Remenyi et al., 2000: 29-30, 152-153)

From the figure above we can observe that unquantifiable, intangible benefits are
those which are the most difficult to measure, assuming that some form of measurement is
the goal of management and other decision makers. However, this does not imply that these
areas of intangible IT benefits need to be excluded from evaluation. Likewise in the ICTD
domain we are hard pressed to extend evaluation frameworks of quantifiable data that are
more easily measured such as teledensity, bandwidth per capita, number of connection points,
number of training certificates issued, etc. Taylor & Zhang (2007) hone in on the issue by
arguing that “when a technology is regarded as the prime initiator of change in society,
measuring the changing technology might seem to be enough” and that “measuring
computers, cables, and connections tells us very little about the actual state of society”.
Let us now turn to the evaluation of ICT in development contexts. In doing so we
draw some lessons from the business world in order to help move the field from the early
euphoria, subsequent instrumental focus on primarily tangible, economic benefits of ICT on
society, towards a more “mature” evaluation perspective which incorporates intangible,
unquantifiable impacts of ICT in development contexts.

5. EVALUATION OF ICT IN DEVELOPMENT CONTEXTS

Although we have developed a substantive body of knowledge regarding the evaluation of
ICTs in business contexts, we have yet to understand how the extant research outputs can be

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 6

applied in ICTD contexts. Researchers, studying business benefits of ICTs, have over the
years adapted theories from other disciplines in their quest to develop models for evaluating
ICT benefits. Examples of these include Communications Theory (Shannon & Weaver,
1949), Resource-Based Theory (Penrose, 1959), and Theory of Reasoned Action (Fishbein &
Ajzen, 1975). However, even the business information systems researchers have been
challenged. This is evident in the fragmented body of ICT effectiveness knowledge and lack
of consistent trends in the application of the associated theoretical paradigms. Moreover the
tools and techniques to evaluate ICTs in the business environment are not useful in the
context of societal development. This therefore exacerbates the problem, as existing theories
and models need to be studied carefully and adapted, if this is indeed possible, for application
in the context of ICTD evaluation. Alternatively new theories for ICT evaluation needs to be
developed. This of course has been recognized by the scientific community, and there have
been a number of calls to the ICTD research community to respond to the challenge by
providing new theories and methods e.g. van Dijk (2006); Heeks (2006); Pather & Uys
(2010). Perhaps, more importantly, we require theories which will enhance a deeper
understanding of the effects of ICTs on social outcomes and the level of success thereof.
Perhaps then only will assessments of economic outcomes derived from ICTs be more
meaningful.
A new approach to ICT evaluation in a development context would also be
convergent with transformations in the notion of development itself. While early notions of
development simply equated it with economic growth and transfer of technology from
developed countries to underdeveloped ones, theories of development have long abandoned
such simplistic and mechanistic approaches in favor of a more holistic view. Such a view
includes meeting basic needs in an endogenous process that builds participatory democracy,
strengthens self-reliance, promotes structural changes, and fosters empowerment and
liberation (Servaes, 2008; Melkote, 2001). But the changes in approaches and theories of
development do not seem to have affected the field of ICTD, which seems to be still in the
development as modernization paradigm.
Today, thirty years after the ICT productivity paradox was highlighted, the challenges
faced in evaluating impact and productivity are as relevant in the ICTD context as they were
in the business environments back then. This underscores the difficulties associated with
measuring ICTs regardless of the area of application and not least of which relates to the
issue of impact measurement. The ‘productivity’ resulting from ICTs in respect of facilitating
socio- economic development of the large numbers of impoverished and underserved
communities are still to be properly understood. This is underscored by Heeks (2008:27),
who in evaluating the initial ICTD era argues that insofar as evaluation is concerned the work
in this field was “held aloft by hype and uncorroborated stories, which fostered a new
interest in objective impact evaluation”.
Thus, in the ICTD environment the same questions that confronted businesses for
many years still prevail, especially since the digital divide is a phenomenon linked not only to
the topic of access to the Internet, but also intrinsically to the one of usage and usage benefit
(Fuchs & Horak, 2008). Even though millions of dollars have been spent on ICTs by donor
and government agencies around the world, we still do not have sufficient insight into
appropriate methods for evaluating the effectiveness of these technologies on especially
socio-economic development. Many ICTD studies and commissioned research by
governments tend to focus on quantitative data in respect of penetration and rates of usage
and adoption. However the true value of social development is not easy to conceptualize and
hence measure. Parthasarathy & Srinivasan (2006) make a strong case that by relying on easy
to measure data for well-defined indicators, econometric techniques may well suffice to
measure development, but development also leads to changes that are not economic and not

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 7

all social changes lend themselves to measurement using well-defined indicators. Qureshi
(2005) is also critical of macroeconomics models used by International Agencies to predict
the effects of government policies relating to information technology investments and
services on economic growth. She argues that while these models play a pivotal role in
decision making, they often cannot explain why certain IT policies do not have the effects
intended of them, or why certain investments in IT infrastructure do not bring about social
and economic change. Taylor & Zhang (2007) lend support to this critique when they assert
that ICTs do not create the transformations in society by themselves; they are designed and
implemented by people in their social, economic, and technological contexts.

6. TANGIBLE IMPACTS OF ICT FOR DEVELOPMENT

There are no dissenting voices in respect of the potential of ICTs to support social
development and numerous interventions have been deployed around the world with the goal
of bridging the so-called “digital divide”. Governments and various non-government and
private, for-profit organizations have jointly invested significantly in this effort. Heeks
(2009b) , for example, in examining a “very broad notion” of ICTD estimates that as of 2007
a total of US$840 billion has been invested in all developing and transitional economies, and
approximately US$57 billion in low-income countries (GNI <US$935). Notwithstanding
such astronomical figures of investment, well articulated evaluation frameworks, spanning
both tangible and intangible impacts, are still a major stumbling block common across
developed and developing countries.
A variety of factors account for this problem, including the sheer complexity of the
task of measuring the impact of information on development, the lack of appropriate impact
indicators, and the difficulty to collect relevant data. Ten years ago we showed that “until
relevant methodologies and adequate tools are developed to effectively assess the social
impact of the application of ICTs for sustainable development from the user’s perspective,
efforts to demonstrate how people are empowered by knowledge will lack credibility”
(Gomez, Hunt & Lamoureux, 1999). The evaluation problems were also corroborated by
others, such as Batchelor et al. (2003) whom, in reporting in a study of seventeen of the
World Bank’s InfoDev ICTD projects found that:

“…analysts and decision makers are still struggling to make sense of the
mixed experience of information technologies in developing countries. Very
often such experiences seem to amount to little more than a heterogeneous
and unrelated set of anecdotes. However spectacular, successful, moving, or
important some of these anecdotes may be, they remain a precarious basis for
justifying major policy or investment decisions”.

Indeed there can be no doubt that we have witnessed steady, albeit slow, progress
insofar as closing the digital divide from the perspective of broad access to ICT. For example,
ITU statistics show that over the last 10 years, the digital divide between the developing and
the developed countries has been narrowing in terms of fixed telephone lines, mobile
subscribers, and Internet users. In contrast to the slow fixed line growth, phenomenal growth
rates in the mobile sector particularly have been able to reduce the gap that separates the
developed from the developing countries from 27 in 1994, to 4 in 2004. The fixed line gap
has been reduced from 11 to 4 during the same period (ITU, 2006).
Nonetheless, more infrastructure and more access does not necessarily translate into
more meaningful use of ICT, and the evidence of its contribution to social and economic
development is still limited. International agencies have explored innovative ways to measure

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 8

the success of their initiatives and their impact on human development with different degrees
of success. Let us briefly discuss some of the most salient experiences in this regard:
World Bank: The World Bank and its affiliates, the International
Telecommunications Union (ITU) and the OECD, have tended to focus on statistical
indicators of economic growth and human development indices. The World Bank’s recent
studies focus on the deployment of broadband connectivity and mobile telephony, and its
contribution to businesses and government applications (World Bank, 2009 ICTD Report,
Extending Reach and Increasing Impact).
The World Bank’s flagship program on ICTD, InfoDev, has focused on drawing
implementation lessons from its many pilot projects around the world and developed a
framework for monitoring and evaluation of pilot projects on ICTD (Batchelor & Norrish,
undated).
International Donors: Several international donors such as Canadian, Swiss, Danish
and Swedish International Development agencies (CIDA, SDC, Danida and Sida,
respectively) and international agencies (UNESCO in particular) have followed broadly the
same direction of InfoDev and other World Bank program directions described above.
Agencies such as the Canadian International Development Research Center (IDRC), on the
other hand, have increasingly moved away from a focus on impact to that of outcomes in
evaluating development activities (Earl, Carden & Smutylo, 2001). The shift is significant:
rather than trying to assess large-scale social impacts, which happen slowly over time and are
the result of what many people do, outcome mapping focuses on assessing the direct and
observable changes in behavior that can be directly attributed to the activities that are
implemented as part of a development program. The shift from evaluating impact to mapping
outcomes comes at a cost. The detailed process to effectively conduct outcome mapping is
extremely time consuming, and the resulting findings offer limited explanation of the larger,
broader, or longer-term benefits social development is expected to produce.
The evaluation of ICTD activities has been caught at the crossroads of these three
trends. The statistics and tangible indicators for economic growth, the lessons learned from
pilot studies or scattered anecdotes of success, and the detailed documentation of outputs and
outcomes do not add up to an understanding of larger social or economic impacts, which are
essential to inform better policy decisions and program implementation.

7. A WAY FORWARD: ASSESSING INTANGIBLES IN ICTD

A fourth trend is emerging in the ICTD evaluation field. This trend is demonstrative of
evaluation frameworks which focus less on direct and tangible economic, social, or political
benefits and more on indirect, intangible benefits. These are more difficult to measure and
point to deeper and more pervasive effects on individuals and communities.
After 20 years of euphoria, and after 10 years of seeking out tangible, quantifiable and
instrumental impacts of ICT for development, we are still missing the boat. We are still
unable to account for the impacts (both positive and negative) of ICT for human
development. It is time therefore to start building a framework to inform new evaluation
paradigms which evaluate the intangible, non-instrumental, unquantifiable impacts of ICT in
development contexts.
Drawing a parallel with the business models discussed earlier, in the field of ICT for
development we appear to be still in the early phases of measuring tangible, quantifiable
benefits (in Zuboff’s model, the Automate phase,measuring technical aspects of ICT) and
Informate phase (measuring ICT production and project implementation), but with little
inroads into an equivalent of the Transformate phase seen in the business world: such an
approach would focus on intangible, unquantifiable benefits as well as the more easily

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 9

measured tangible, quantifiable benefits. What would the intangible benefits of ICT for
development look like? In Figure 3 we offer a preliminary characterization:

Phases of Business IT ICT and Development

Impact (based on Zuboff, 1988)
Evaluation
Early: Automate: measure Euphoria: quantifiable outputs
quantifiable, technical aspects of IT (rate of (number of computers, number of
tangible information flow, accuracy, users, rate of bandwidth
outputs timeliness) consumption; teledensity; e-
literacy rate; etc)
Intermediate: Informate: IT production Instrumental: economic
measurable, and project implementation outcomes (income generation,
mostly tangible business opportunities, etc)
outcomes
Mature: Transformate: service Intangibles: aspirational
unquantifiable, perspective, intangible benefits
outcomes (empowerment, self
intangible (trust, loyalty, brand, etc) esteem & self worth; social
impacts cohesion & social fabric); citizen
empowerment (relationship with
governments); Individualised
motivator factors (achievement,
recognition).
Figure 3: Phases of ICT Evaluation in Business and in Development

There are indications of a growing awareness of the need to explore intangible and
unquantifiable impacts, even among the international organizations that have promoted the
early focus on tangible and measurable outputs and outcomes. For example, the World
Telecommunication and ICT Development report (ITU, 2006) stated that:

“Instead, the most important economic impact of the spread and use of ICTs
is indirect, by transforming the way individuals, businesses and other parts of
the society work, communicate, and interact. The beneficial impact of ICTs
on productivity – which can help reduce poverty – is of particular interest as
ICT diffusion levels across all countries rise. One way of understanding the
difficulty of measuring the impact that ICTs have, is to imagine the impact
that electricity has had on the economy and society. As with ICTs, there is no
denying that electricity has had important impacts on individuals, businesses
and society at large but its measurement is elusive. Part of the difficulty is that
both ICTs and electricity are “enabling” or “General Purpose Technologies”,
which means their use and their impacts are ubiquitous yet difficult to
measure because they are mainly indirect. It is not electricity or ICTs as such
that make the (bulk) impact on economy and society but how they are used to
transform organisation, processes and behaviours”.

The foregoing endorses our views regarding the “elusive” and “ubiquitous” nature of
ICTD impact measurement, and also highlights the importance of focusing on the “enabling”
and “transformational” aspects in our evaluation frameworks. McNamara (2003) extends this
point in the following excerpt, emphasizing the enabling role of ICTs in one of these elusive
areas impact viz. that of social capital:

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 10

There is increasing evidence that a dense and complex layer of social

institutions, formal and informal groups, and networks of interaction and
common interest between the individual citizen and the state is good both for
the stability and responsiveness of the political system and for the economy
and society as a whole. This social capital enables richer and more diverse
views to surface on important societal issues, empowers groups to address
common concerns and interests without necessarily relying on government
intervention, and can even help the emergence of social consensus by
permitting multidirectional debate and sharing of information among those
with different perspectives through mechanisms that are not directly tied to
the formal political and governmental structure. By facilitating new forms of
many-to-many communication, collaboration, and information-sharing, both
within a given country and among groups with similar interests and concerns
across borders, ICTs can add to the vibrancy of civil society institutions and
networks as a check on government, a source of ideas and innovations, and an
outlet for the interests, concerns and desires for solidarity on the part of
individuals and groups. (McNamara, 2003)

A question worthy of introspection is “Why have these calls for attention to intangible
impacts, from influential people and organizations, fallen on deaf ears?”. Our premise is that
this lack of response is simply because evaluating intangible impacts is far more challenging,
and hence we have chosen not to go down the path of complexity and murkiness, and our
baskets of tangible and quantifiable indicators remain as they are, incomplete as they may be.
Similar evaluation problems have been faced before, e.g. evaluating benefits of water supply
and electricity. One such study highlights the intangible aspect of empowerment. In this study
it was found that given that the poor often feel marginalized and their lives can be
characterized by a sense of powerlessness and instability, even the perception of benefits can
assist in empowerment, which is likely to lead to proactive initiatives by the poor themselves
(Songco, 2002). Another example of an intangible impact, referred to by McNamara above is
that of cohesiveness. In the words of Reimer, only cohesive people can “respond collectively
to achieve valued outcomes and to deal with the economic, social, political, or environmental
stresses (positive or negative) that affect them” (Reimer, 2002). This therefore suggests that
the tangible impacts that we have been so focused on are in fact facilitated and preceded by
the intangible impacts on which we have been focused. Thus we have expended our energies
on evaluating impacts which are too far down the impact timeline, and this has contributed to
the fuzziness which belies the linking of ICTs with development.
We suggest therefore that examples of intangible impacts which need to be evaluated
could include empowerment, self-esteem, and sense of self-worth, at the individual level, and
social cohesion and strengthening of social fabric, at the collective level. Social cohesion was
defined by Ritzen and colleagues as “a state of affairs in which a group of people demonstrate
an aptitude for collaboration that produces a climate for change” (Ritzen et al, 2000).
According to these authors, the role of social cohesion is to strengthen the trust inside and
between different social groups in order to accelerate socio-economic growth with “a
reasonable distribution over those groups in the population”. In the same way, Maxwell
(1996) identified social cohesion with the processes of building shared values and
communities of interpretation, reducing disparities in wealth and income, enabling people to
have a sense that they are engaged in a common enterprise, facing shared challenges, and that
they are members of the same community. These are all impacts that have been anecdotally

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 11

reported in the ICTD literature, but with little traction in the face of the dominant attention
paid to tangible and quantifiable, economic impacts.
Let us take the case of a small semi-rural fishing village in the southernmost tip of
Africa. None of the fisherman in this village have actually learned how to use the public
access computer at the local library, a typical community based ICTD project2. However
evidence from a case study (Pather & Mitrovic, 2008) indicates that the public access
computers in the library provided a means of uniting the fishermen around a common cause
related to fishing licenses. This in turn improved their basic livelihoods in indirect ways. If
the evaluation of the centre was based on tangible measures of usage statistics, such an
impact would not have been identified. In this case, a closer look at issues of empowerment
and cohesion provided evidence that ICTs have indeed transformed the life of the community
in ways that are far more important than what the number of users of computers would
reveal.

8. CONCLUSION
We require new measurement strategies and a shift in focus from measuring the tangible
symptoms or outputs to measuring intangible outcomes e.g., the more strategic development
goals such as improved self-reliance and social cohesion. These are important building blocks
of socio-economic development, and thus our ICT evaluation toolkits need to account for
such indicators. Currently agencies and governments find it difficult to rationalize their
expenditure and evidence. At most, much of the evaluation done to date is based on
quantitative output. Some agencies, though not all, are able to provide data such as the
number of people trained or rate of usage of centers. However this in itself does not inform us
if any meaningful change has been brought about in people’s lives. An important question
that should therefore be placed on the ICTD research agenda is: How do we extract credible
evidence of the socio-economic change in people’s lives, especially when this change is
unquantifiable and intangible? A further question that should also be placed on the agenda is:
what are useful techniques for evaluating such change? The value of ICTD evaluation lies in
producing data that substantiates socio-economic outcomes and impact. The research and
practitioner community has to work hard to ensure that as the next era, viz. ICTD 2.0 (see
Heeks, 2008) dawns upon us we should no more be held aloft by hype and uncorroborated
stories.
The example of the fishing village described in the foregoing section is a typical case
of an anecdote which does not convincingly provide evidence of impact due to a lack of a
suitable framework which goes beyond tangible and quantifiable indicators of success of ICT
for development projects. A collection of numerous anecdotes such as this does not, in itself,
constitute evidence of impact. Instead, a structured approach to evaluate impact of ICT,
which includes attention to intangible and unquantifiable aspects (e.g. self-worth and the
strengthening of social fabric) that are facilitated indirectly through use, or even the presence
of ICTs, will provide a more complete and holistic perspective of ICT impact.
Finally, we discussed how the euphoria and disillusionment with ICT for development
may be a consequence of two issues. Firstly, it stems from its roots in a modernization
paradigm which equates development with economic growth. Secondly it has to do with an
evaluation approach that focuses on measuring quantifiable, tangible outputs and outcomes of
the ICTD interventions. Drawing from the experience of ICT evaluation in the business
environment we suggest that it is not only possible but desirable to mature into an evaluation
paradigm which includes attention to intangible and unquantifiable impacts of ICT. Thus a
focus on empowerment and social fabric instead on economic growth would also be
                                                            
2
For an overview of the Cape Access Project see http://www2.capeaccess.org.za/ 

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 12

convergent with current theories and approaches to development, which have long abandoned
the modernization paradigm. To this end, all stakeholders in the ICTD environment,
including researchers, practitioners, and funders need to undergo a process of
conscientization. In so doing we should aim to detach ourselves from the fixation with
narrow tangible and quantitative impact metrics. A more detailed exploration of the
theoretical and methodological implications of this shift in the ICT for development field is
warranted.

9. REFERENCES
Bailey, J.E. and Pearson, S.W. (1983) Development of a Tool for Measuring and Analyzing
Computer User Satisfaction, Management Science, 29, 5, 530-545.
Bannister, F., Berghout, E., Griffiths, P. and Remenyi, D. (2006) Tracing the Eclectic (or
maybe even Chaotic) Nature of ICT Evaluation, in Remenyi, D. & Brown, A. (Eds).
Proceedings of the 13th European Conference on Information Technology Evaluation,
Genoa, Italy, 28-29 September. Reading: Academic Conferences: 41-51.
Batchelor, S., Evangelista, S., Hearn, S., Peirce, M., Sugden, S. and Webb, M. (2003) ICT
For Development: Contributing to the Millennium Development Goals – Lessons
Learnt from Seventeen infoDev projects, Washington DC: World Bank.
Batchelor, S. & Norrish, P. (undated) Framework for the Assessment of ICT Pilot Projects:
Beyond Monitoring and Evaluation to Applied Research. InfoDev, World Bank,
Washington DC. http://www.infodev.org/en/Publication.4.html
Boyd, D.F. and Carson, H.H.J. (1963) Economic Evaluation of Management Information
Systems, IBM Systems Journal, 2, 2-23.a
Brynjolfsson, E. (1993) The Productivity Paradox of Information Technology,
Communications of the ACM, 36, 12, 66-76.
DeLone, W.H. and McLean, E.R. (1992) Information Systems Success: The Quest for the
Independent Variable, Information Systems Research, 3, 1, 60-95.
Earl,S., Carden, F. and Smutylo, T. (2001) Outcome Mapping: Building Learning and
Reflection into Development Programs, Ottawa: IDRC.
Fishbein, M. and I. Ajzen. (1975) Belief, Attitude, Intention and Behaviour: An Introduction
to Theory and Research, Reading: Addison-Wesley.
Fuchs, C., and Horak, E. (2008) Africa and the Digital Divide, Telematics and Informatics,
25, 2, 99-116.
Gallagher, C.A. (1974) Perceptions of the Value of a Management Information System,
Academy of Management Journal, 17, 46-55.
Gomez, R., Hunt, P., and Lamoureux, E. (1999) Telecentre Evaluation and Research: A
Global Perspective, in Gomez, R. & Hunt, P. (Eds.), Telecentre Evaluation: A Global
Perspective, Ottawa: IDRC, 13-29.
Gore, A. (1994) Address to the International Telecommunication Union, 21 March.
Heeks, R. (2006) Theorizing ICT4D Research, Information Technologies and International
Development, 3, 3, 1-4.
Heeks, R. (2008) ICT4D 2.0: The Next Phase of Applying ICT for International
Development, Computer, 41, 6, 26-33.
Heeks, R. (2009a) The ICT4D 2.0 Manifesto: Where Next for ICTs and International
Development? Working paper series, Development Informatics Group, Institute for
Development Policy and Management, Manchester.
Heeks, R. (2009b) Worldwide Expenditure on ICT4D, ICTs for Development Talking about
Information and Communication Technologies and Socio-economic Development,
Blog, http://ict4dblog.wordpress.com/2009/04/06/worldwide-expenditure-on-ict4d/

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 13

Hitt, L.M. and Brynjolfsson, E. (1996) Productivity, Business Profitability, and Consumer
Surplus: Three Different Measures of Information Technology Value, MIS Quarterly,
20, 2, 121-142.
Irani, Z., Love, P.E.D. and Zairi, M. (2000) Information Systems Evaluation: Mini-Track
Introduction, in Chung, M. (ed.). Proceedings of the 6th Americas Conference on
Information Systems, Long Beach, California, 10-13 August, Long Beach, CA:
California State University, 1073-1075.
ITU (2006) World Telecommunication/ICT Development Report 2006 - Measuring ICT for
Social and Economic Development, Executive Summary. Geneva: International
Telecommunications Union. http://www.itu.int/dms_pub/itu-d/opb/ind/D-IND-
WTDR-2006-SUM-PDF-E.pdf
Jiang, J.J., Klein, G. and Carr, C.L. (2002) Measuring Information System Service Quality:
SERVQUAL from the Other Side, MIS Quarterly, 26, 2, 145-166.
Joshi, K. and Pant, S. (2002) A Classification Framework to Assess and Guide IT
Investments, in Banker, R.D., Chang, H. and Kao, Y-C. (Eds), Proceedings of the 8th
Americas Conference on Information Systems, Dallas, Texas, 9-11 August, Atlanta,
GA: AIS, 1239-1243.
Kohli, R., Sherer, S.A. and Baron, A. (2003) Editorial – IT Investment Payoff in E-business
Environments: Research Issues. Information Systems Frontiers, 5, 3, 239-247.
Laudon, K.C. and Laudon, J.P. (2000) Management Information Systems: Organization and
Technology in the Networked Enterprise. 6th ed. Upper Saddle River, NJ: Prentice
Hall.
Lomerson, W.L. and Tuten, P.M. (2005) Examining Evaluation across the IT Value Chain,
Proceedings of the 8th Annual Conference of the Southern Association of Information
Systems, Savannah, Georgia, 25-26 February: 124-129.
Lucas, H.C. (1973) User Reactions and the Management of Information Services,
Management Informatics, 2, 4, 165-172.
Mansell, R. & Wehn, U. (eds) (1998) Knowledge Societies: Information Technology for
Sustainable Development. United Nations Commission on Science and Technology
for Development. New York: Oxford University Press.
Maxwell, J. (1996) Social Dimensions of Economic Growth. Eris John Hanson Memorial
Lecture Series, Vol. VIII, Alberta: University of Alberta.
McNamara, K.S. (2003) Information and Communication Technologies, Poverty and
Development: Learning from Experience. InfoDev, Washington, DC: World Bank.
Melkote, S. (2001) Communication for Development in the Third World. New Delhi: Sage.
O'Neil, D. (2002) Assessing Community Informatics: A Review of Methodological
Approaches for Evaluating Community Networks and Community Technology
Centers, Internet Research: Electronic Networking Applications and Policy, 12, 1,
76-102.
Parthasarathy, B. and Srinivasan, J. (2006) Innovation and its Social Impacts: The Role of
Ethnography in the Evaluation and Assessment of ICTD Projects, Global Network for
Economics of Learning, Innovation, and Competence Building Systems Conference,
4-7 October, Trivandrum, India.
Pather, S. and Mitrovic, Z. (2008) ICTs for Participatory Local Development in South Africa:
Exploring a Systemic Approach, Bangalore, India: IT for Change.
Pather, S. and Uys, C. (2010) A Strategy for Evaluating Socio-Economic Outcomes of an
ICT4D Programme, Proceedings of the 43rd Hawaii International Conference on
System Sciences (HICSS-43), IEEE Computer Society.
Penrose, E.T. (1959) The Theory of the Growth of the Firm. New York: John Wiley.

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org 
EJISDC (2012) 50, 5, 1-14 14

Qureshi, S. (2005) How Does Information Technology Affect Development? Integrating

Theory and Practice into a Process Model, Proceedings of the 11th Americas
Conference on Information Systems, Omaha, USA, August 11-14, 500-509.
Reimer, B. (2002) Understanding and Measuring Social Capital and Social Cohesion,
Montréal: Canadian Rural Restructuring Foundation, Concordia University.
Remenyi, D., Grant, K., and Pather, S. (2004) It was a Shock when Boo Went under: the
Legacy of the E-bubble: Lessons for Managers, Journal of General Management, 29,
3, 24-36.
Remenyi, D., Money, A.H., Sherwood-Smith, M. and Irani, Z. (2000) The Effective
Measurement and Management of IT Costs and Benefits. 2nd Ed. Oxford: Elsevier
Butterworth-Heinemann.
Ritzen, J., Easterly, W. and Woolcock, M. (2000) On “Good” Politicians and “Bad” Policies:
Social Cohesion, Institutions, and Growth. World Bank Policy Research Working
Paper 2448, Washington DC: World Bank.
Seddon, P.B., Staples, S., Patnayakuni, R. and Bowtell, M. (1999) Dimensions of Information
Systems Success, Communications of the Association for Information Systems, 2, 20,
1-60.
Servaes, J. (2008) Communication for Development and Social Change, New Delhi: Sage.
Shannon, C.E. and W. Weaver. (1949) The Mathematical Theory of Communication, Urbana,
IL: University of Illinois Press.
Shaw, N.C., DeLone, W.H. and Niederman, F. (2002) Sources of Dissatisfaction in End-user
Support: An Empirical Study, ACM SIGMIS Database, 33, 2, 41-56.
Solow, R.M. (1987) We’d Better Watch Out. New York Times Book Review 36, July 12.
Songco, J.A. (2002) Do Rural Infrastructure Investments Benefit the Poor? Evaluating
Linkages: A Global View, A Focus on Vietnam, Washington DC: World Bank.
Srinivasan, A. (1985) Alternative Measures of System Effectiveness: Associations and
Implications, MIS Quarterly, 9, 3, 243-253.
TASCHA (2010) Global Impact Study of Public Access to Information & Communication
Technologies, Technology & Social Change Group, University of Washington.
http://www.globalimpactstudy.org.
Taylor, R. and Zhang. B. (2007) Measuring the Impact of ICT: Theories of Information and
Development, Telecommunications Policy Research Conference, September 26-28,
Washington, D.C. http://www.intramis.net/TPRC_files/TPRC%2008%20Taylor-
Zhang%20Final.pdf
Trice, A. and Treacy, M. (1986) Utilization as a Dependent Variable in MIS Research, in
Maggi, L., Zmud, R. and Wetherbe, J. (eds). Proceedings of the 7th International
Conference on Information Systems (ICIS), San Diego, California, 15-17December,
227-239.
Tuan, M. (2008) Measuring and/or Estimating Social Value Creation: Insights into Eight
Integrated Cost Approaches, Bill & Melinda Gates Foundation, Impact Planning and
Improvement, Seattle.
van Dijk, J.A.G.M. (2006) Digital Divide Research, Achievements and Shortcomings,
Poetics, 34, 221–235.
Whyte, G., Bytheway, A. and Edwards, C. (1997) Understanding User Perceptions of
Information Systems Success, Strategic Information Systems, 6, 35-68.
Wiggers, P., Kok, H. and De Boer-De Wit, M. (2004) IT Performance Management, Oxford:
Elsevier Butterworth-Heinemann.
Zuboff, S. (1988) In the Age of the Smart Machine: The Future of Work and Power. New
York: Basic Books.

The Electronic Journal on Information Systems in Developing Countries

http://www.ejisdc.org