Measuring Firm Innovation
Measuring Firm Innovation
Measuring Firm Innovation
org/18606
DETAILS
274 pages | 8.5 x 11 | HARDBACK
ISBN 978-0-309-38717-0 | DOI 10.17226/18606
CONTRIBUTORS
Robert E. Litan, Andrew W. Wyckoff, and Kaye Husbands Fealing, Editors; Panel on
Developing Science, Technology, and Innovation Indicators for the Future;
BUY THIS BOOK Committee on National Statistics; Division of Behavioral and Social Sciences
and Education; Board on Science, Technology, and Economic Policy; Policy and
Global Affairs; National Research Council
Visit the National Academies Press at nap.edu and login or register to get:
– Access to free PDF downloads of thousands of publications
– 10% off the price of print publications
– Email or social media notifications of new titles related to your interests
– Special offers and discounts
All downloadable National Academies titles are free to be used for personal and/or non-commercial
academic use. Users may also freely post links to our titles on this website; non-commercial academic
users are encouraged to link to the version on this website rather than distribute a downloaded PDF
to ensure that all users are accessing the latest authoritative version of the work. All other uses require
written permission. (Request Permission)
This PDF is protected by copyright and owned by the National Academy of Sciences; unless otherwise
indicated, the National Academy of Sciences retains copyright to all materials in this PDF with all rights
reserved.
Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy
Measuring Innovation
This chapter considers the measurement of innovation— essential that such measures be used in concert with in-
in particular, the outputs of the country’s innovation sys- depth analysis of economic and institutional environments
tem.1 It presents definitions of innovation, explains the and understanding of behavioral responses to changes in
importance of measuring innovation and the relevance of those environments. An innovation may reach forward into
innovation measures for policy decisions, and examines varied markets and backward to human capital and at times
the role of innovation surveys and their limitations. The research and development (R&D) inputs. The discussion in
chapter then reviews improvements to innovation surveys, this chapter must therefore be viewed in the context of an
in particular the Business Research and Development and indicators program that is analytically strong.
Innovation Survey (BRDIS), including improving inter-
national comparability, gathering deeper information on
DEFINITIONS
innovations, extending BRDIS to include such items as
organizational and marketing as well as “unmarketed” Schumpeter (1934, p. 66) provided a definition of “inno-
innovations and to track a broader array of inputs to inno- vation” early in the 20th century. He defined product innova-
vation, improving the presentation of information, and tion as “the introduction of a new good . . . or a new quality
improving linkages between the BRDIS data and other of a good” and process innovation as “the introduction of a
datasets. Finally, the chapter turns to the use of nontradi- new method of production . . . or a new way of handling a
tional methodologies to track innovation, such as the use commodity commercially.” This definition influenced early
of “business practice” data. attempts to measure the activity of innovation through case
In accordance with the framework used in this study studies and surveys. Other definitions offered more recently
(Chapter 2), the panel’s recommendations for the National are presented in Box 4-1.
Center for Science and Engineering Statistics (NCSES) Knowledge gained from a decade of experience with
regarding the measurement of innovation are driven by key experimental surveys in Canada, Germany, the Nordic
policy questions and by specific questions raised by users of countries, and the United States was codified by OECD in
science, technology, and innovation (STI) indicators, as well the first Oslo Manual (OECD, 1992), which dealt with inno-
as by the need for internationally comparable measures. The vation in manufacturing products and processes. Following
panel acknowledges that the innovation measures presented more measurement experience, the scope of the manual was
in this chapter answer policy questions only in part. It is extended to the nonagricultural economy (OECD-Eurostat,
1997). This broadened scope gave rise to a better understand-
1Edquist (2005, p. 182) defines the “system of innovation” as “all im- ing of the measurement of innovation in service industries
portant economic, social, political, organizational, institutional, and other and expansion of the definition of innovation to include not
factors that influence the development, diffusion, and use of innovations.”
Lundvall (1992, p. 2) states that a system of innovation is “constituted by
only product and process innovations but also organizational
elements and relationships which interact in the production, diffusion and change and the development of new or the extension of exist-
use of new, and economically useful, knowledge.” Lundvall (2007, p. 106) ing markets. These additional components of the definition
extends this definition by focusing on the learning processes, stating: “The were also considered by Schumpeter.
analysis of innovation systems may be seen as an analysis of how knowl- For measurement purposes, the Oslo Manual (OECD-
edge evolves through processes of learning and innovation.” Other noted
studies on national innovation systems include Freeman (1995); Furman et
Eurostat, 2005, p. 46) defines innovation as follows:
al. (2002); Lundvall (1988); Martin and Johnston (1999); Mowery (1992);
Muller and Zenker (2001); and Nelson (1993).
41
BOX 4-1
Definitions of Innovation
The following definitions of innovation vary, but the common thread is the extraction of economic value from novel activities (Innovation Vital Signs
Project 2007):
Innovation is “the commercial or industrial application of something new—a new product, process or method of production; a new market or sources
of supply; a new form of commercial business or financial organization.”
Schumpeter 1983
Innovation is the “intersection of invention and insight, leading to the creation of social and economic value.”
Council on Competitiveness 2005
Innovation covers a wide range of activities to improve firm performance, including the implementation of a new or significantly improved product,
service, distribution process, manufacturing process, marketing method or organization method.
European Commission 2004
Innovation—the blend of invention, insight and entrepreneurship that launches growth industries, generates new value and creates high value jobs.
Business Council of New York State 2006
The design, invention, development and/or implementation of new or altered products, services, processes, systems, organization models for the
purpose of creating new value for customers and financial returns for the firm.
Advisory Committee on Measuring Innovation in 21st Century Economy, Department of Commerce 2008
An innovation is the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new
organizational method in business practices, workplace organization or external relations. Innovation activities are all scientific, technological, orga-
nizational, financial and commercial steps which actually, or are intended to, lead to the implementation of innovations.
OECD-Eurostat 2005
Innovation success is the degree to which value is created for customers through enterprises that transform new knowledge and technologies into
profitable products and services for national and global markets. A high rate of innovation in turn contributes to more market creation, economic
growth, job creation, wealth and a higher standard of living.
Innovation Vital Signs Project 2007
such information, which has been shown to be policy relevant. For CIS 2012,
the relevant question is 4.1 (Eurostat, 2012).
MEASURING INNOVATION 43
WHY MEASURE INNOVATION? intellectual property rights. Also, the propensity to patent is
highly industry dependent.
The underlying analytical framework used in this study
Similarly, input measures such as R&D spending cannot
posits that new and improved products and processes, new
adequately substitute for measures of innovation output,
organizational methods, and new marketing concepts and
especially if policy makers are concerned about government
strategies are key measures of the output of a country’s
funds being used efficiently. Moreover, recent evidence
innovation system. Innovation affects both economic per-
shows that most firms that report innovations do not report
formance measures, such as productivity growth, profits,
R&D spending. For example, Table 4-1, drawn from the 2011
and job creation, and noneconomic variables, such as life
BRDIS results, shows that 79 percent of all firms with new or
expectancy and environmental response. Meanwhile, the
significantly improved products or processes did not report
rate of innovation responds to inputs such as R&D spend-
R&D spending. In other words, focusing only on firms that
ing; the availability of science, technology, engineering, and
report R&D spending raises the possibility of missing much
mathematics (STEM) labor; regulatory policies; and other
of the innovation taking place in the economy.4
variables.3
Many innovations developed and introduced by start-up
From this perspective, the panel strongly believes that
companies may not be associated with formal R&D, or R&D
NCSES needs to improve its ability to measure and track
may not be recorded as such because start-ups often have no
innovation. Improved measures of innovation are necessary
revenues against which to record expenses. More broadly,
to assess the impact of federal, state, and local innovation
advances in information and communication technologies
policies, such as the amount and direction of federal R&D
and the importance of innovations such as Google’s search
funding, support for STEM education at the graduate level,
service, Amazon’s e-commerce website, and Apple’s iTunes
and regulation of new products and services. In addition,
store signal how the nature of the innovation process has
having good measures of innovation output facilitates com-
changed, increasingly depending on investments in inno-
parison of the United States with other countries in a key area
vative assets other than R&D (e.g., data, organizational
that promotes economic growth. NCSES’s mandate in the
know-how) and with take-up primarily in the service sector.
America Creating Opportunities to Meaningfully Promote
Indeed, in fiscal year 2006, the year before the iPhone was
Excellence in Technology, Education, and Science (America
introduced, Apple reported spending less than 3 percent
COMPETES) Act (U.S. House of Representatives, 2010)
of its revenues on R&D—relatively low for a technology
includes the curation and dissemination of data on “United
company.5
States competitiveness in science, engineering, technology,
Finally, broad measures of economic performance, such
and research and development.” Innovation is an important
as productivity, do contain information about the country’s
element for such comparisons. Without improved direct indi-
innovation system, especially over longer periods such as
cators of innovation outputs, policy analysis will continue to
decades. However, productivity growth is a weak signal in
rely on imperfect indicators of innovation, such as number
the short run because it is affected by external factors such
of patents granted; inputs to innovative activities, such as
as the business cycle and by external factors such as the
R&D spending and number of STEM workers; and broad
availability of finance.
performance measures for the economy, such as productivity.
The panel acknowledges that developing useful measures
A voluminous literature addresses the historical link
of innovation output is far more difficult than measuring
between patents and innovation, and the topic remains con-
inputs or patents, for example. The space of economically
troversial today. However, there is little doubt that the more
significant innovations is both diverse and expanding. Inno-
than doubling of patent grants over the past 20 years—from
96,511 in 1991 to 224,505 in 2011—did not reflect an equally 4Sectors in which more than 50 percent of firms indicated that they were
sharp rise in the volume of innovation. As recent litigation
engaged in either product or process innovation between 2009 and 2011
has shown, multiple patents have become a bargaining chip were manufacturing—petroleum and coal products; chemicals (particularly
rather than an accurate measure of innovation output. More- basic chemicals; soap cleaning compound, and toilet preparation; paint,
over, a recent NCSES InfoBrief suggests that patents are coating, adhesive, and other chemicals); machinery (particularly agricul-
less important to companies than other forms of intellectual tural implement; engine, turbine, and power transmission equipment);
computer and electronic products (particularly communications equipment;
property protection (Jankowski, 2012). Based on data from
semiconductor and other electronic components; navigational, measuring,
the 2008 BRDIS, firms across virtually every industry are electromedical, and control instruments; other measuring and controlling
much more likely to cite trademarks, copyrights, and trade devices); and transportation equipment (particularly guided missile, space
secrets than either design or utility patents as important for vehicle, and related parts); or nonmanufacturing—information (particularly
software publishers). See Appendixes H-K for specific details.
5Apple’s R&D/sales ratio has historically been relatively low compared
3One prominent model is presented by Crépon and colleagues (1998). with companies such as Google, Microsoft, Intel, and Samsung. For ex-
This three-equation model has the following structure: (1) RD = f (Xr); ample, Apple’s R&D/sales ratio was at 2.27 percent in fiscal year 2012,
(2) Innovation = g(RD, Xi); and (3) Productivity/other economic output approximately one-third of Samsung’s R&D/sales ratio. However, Apple
= h(Innovation, Xp), where X’s are vectors of other factors relevant to the may have benefited from R&D spending by suppliers such as Samsung
output (see also Mairesse and Mohnen, 2010; OECD, 2009). and Intel.
TABLE 4-1 Research and Development (R&D) Firms Are More Likely to Innovate, but Most Innovating Firms Do Not Do
R&D
% of Firms with New or Significantly Improved Products or Processes % of Total Firms in Scope*
R&D Status of Firm Doing R&D Not Doing R&D Total
Firms Doing R&D 64 N.A. 21 5
Firms Not Doing R&D N.A. 12 79 95
NOTE: A firm that does R&D may produce many innovations, while a firm without R&D may produce only one innovation. However, BRDIS does not
report on the number of innovations per firm.
*Approximately 1.2 million firms on a weighted basis are in scope, that is, reported to the survey.
SOURCE: Calculated by the panel from the 2011 BRDIS, Table 49; see Appendix K. For more information on innovation statistics from the 2011 cycle
of BRDIS, see Business R&D and Innovation: 2011, which will be published in 2014. For previous cycles, the full set of detailed statistical tables is available
in Business R&D and Innovation: 2008-2010 (NSF 13-332) at http://www.nsf.gov/statistics/nsf13332/start.cfm [April 2014]. Innovation statistics from the
2008 BRDIS are found in Tables 33-36, the 2009 BRDIS innovation statistics are found in Tables 82-85, and the 2010 BRDIS innovation statistics are found
in Tables 126-129. Relative standard errors are available from NCSES staff upon request.
vation can include everything from the successful develop- such indicators could help address. This is not to say that the
ment and approval of gene therapy, to the development of the indicators suggested here are relevant only for government
special high-strength glass used for smartphone screens, to policy makers; many of them should be useful to university
the steady improvement of unmanned aerial vehicles used by administrators and business managers as well.
the military in Afghanistan, to logistical warehouse manage- As noted above, the value of a measure of innovation is
ment systems. Other significant contemporary innovations that it offers the possibility of a direct link between innova-
include Google’s AdSense/AdWords technology for targeted tion outcomes and relevant policy variables, many of which
online advertising and Apple’s invention of the App Store are innovation inputs. In particular, one or more innovation
concept, which enables small programming firms to sell and measure can make it possible to test the impact on innovation
get paid for lightweight mobile applications. Although some of such factors as R&D spending, the availability of STEM
may disagree, moreover, the creation of credit default swaps labor, regulation, market power, globalization of research,
in the 1990s was almost certainly an economically significant immigration policy, and tax policy (see Box 4-2). Note that
innovation of a type that should be considered by NCSES.6 these factors can have either a positive or a negative effect
Nevertheless, it is clear that as a national goal, policies on innovation, and it is essential for policy makers to know
that encourage bringing more innovations to market are use- which way the arrow points and under what conditions.
ful if they generate economic growth and jobs and improve Firm-level analysis also links investment in innovation to
the nation’s competitiveness. As discussed below, measures improved outcomes. Using the third Community Innovation
of innovation outputs are needed, and providing such mea- Survey (CIS3) for 16 European countries, Jaumotte and Pain
sures is clearly within the purview of NCSES to support (2005) found that the proportion of firms engaged in inno-
such policies. vation spending was closely correlated with the proportion
of successful innovators in a country. Their results suggest
RECOMMENDATION 4-1: The National Center for that a 1 percentage point increase in aggregate innovation
Science and Engineering Statistics should develop spending was associated with an increase of 0.85 percent-
additional indicators for measuring innovation age points in the probability of being a successful innovator
outcomes that would complement existing data on and an increase of 0.7 percentage points in the share of new
patents, inputs to innovation activities, and broader products in turnover.7
measures of economic performance. An analysis of innovative companies could shed light on
such key policy questions as the appropriate amount and
direction of federal R&D funding, limits on or encourage-
POLICY RELEVANCE OF INNOVATION MEASURES
ment of immigration of skilled workers, support for STEM
Information should not be collected in a vacuum. It is
good business practice for NCSES to focus on developing 7In this study, innovation spending included not only intramural R&D but
new indicators that are useful for informing policy decisions, also extramural R&D, other acquisitions of knowledge (such as rights to use
especially given the current fiscal environment. This section patents, licenses, and software), investment in capital goods, expenditures
on training, and spending necessary for the innovation to be placed on the
identifies some innovation-relevant policy questions that
market (for example, marketing or design expenditures). Non-R&D expen-
ditures were sometimes more important than those on R&D, and many of the
6According to one account, the concept of credit default swaps was countries with the highest proportion of successful innovators also had the
developed in 1994 to help Exxon fund its huge potential liability for the highest propensity to engage in non-R&D innovation spending—showing
Exxon Valdez disaster (see Tett, 2009). the importance of taking a broad view of innovation inputs.
MEASURING INNOVATION 45
of expectations. These are potentially high-return areas for January 2009. See National Science Foundation (2013a) for the BRDIS
policy, where relatively small policy changes could poten- questionnaire.
9It is widely known that the innovation statistics from BRDIS and the
tially lead to much better results.
CIS lack comparability; see an explanation in Hall (2011). Also see the
The most striking potential bottleneck is in biomedical discussion of the disparities between the U.S. and European innovation
research, with the pharmaceutical industry and the National statistics later in this chapter.
BOX 4-3
Innovation Questions on the 2011 Business Research and
Development and Innovation Survey (BRDIS)
MEASURING INNOVATION 47
Department of Commerce, 2012), which contained sections evidence to assess the factors that may drive biases in inter-
on: the 2011 survey was in the field as of this writing (see national comparisons. (See Table 4-2 for a list of websites for
Box 4-3 for selected innovation questions from the 2011 innovation surveys from regions around the world.)
survey). The 2011 version of the BRDIS (U.S. Department Complete cognitive testing of the questions used on
of Commerce, 2012) contained sections on: innovation surveys is ongoing in the United States and
Europe. Nevertheless, the data are useful for preliminary
• innovation, reported by the nature of the innovation snapshots. OECD-NESTI has been conducting a cognitive
(new product or new process), and the percentage of testing research project since 2011, with participation by
sales coming from newly introduced products; representatives from the United States and other countries.
• R&D spending, reported by business code, country, The project is examining how businesses define and mea-
state, funder, and performer; sure innovation. This effort is motivated by the sizable gap
• characteristics of R&D spending, reported by type between estimates in the United States and Europe on the
(basic versus applied versus development) and area incidence of innovation among firms, with 2008 BRDIS data
of focus (e.g., biotechnology, nanotechnology); being used as the baseline for U.S. innovation statistics. The
• R&D employees, by type (scientists and engineers 2009 and 2010 BRDIS instruments included some systematic
versus technicians and technologists); and variations aimed at determining whether the way innovation
• use of intellectual property, including patents, and questions were being asked substantially influenced the
trademarks. incidence answers provided. During a discussion with panel
members, NCSES staff indicated that the conclusion to date
NCSES’s efforts to revise BRDIS are aimed at gathering is that there is no major difference in the statistics due to the
more information on innovation activities, going beyond phrasing of the questions. The weak result on innovation
simple “yes/no” questions on whether a firm introduced or changed little when the questions were moved to the front of
significantly improved goods, services, or processes. These the 2009 BRDIS (see Table 4-3). The relatively low reported
efforts also are aimed at enhancing comparability with key incidence of innovation in the United States compared with
questions in the CIS and ensuring that the questions on European nations remains an issue for further research by
innovation are answered by firms that do not conduct R&D. NCSES and its partners in OECD-NESTI.10 In addition, the
Comparability of the BRDIS and CIS data also depends on size dependence of the propensity to innovate may contribute
surveying similar populations of firms and on the techniques to the weak results, suggesting that the Table 4-3 should be
used to derive estimates from the data. produced for at least three employment cutoffs (see Recom-
As befits a cutting-edge survey, BRDIS is still evolving. mendation 4-2 later in this chapter).
It is important to recognize that such concepts as invention,
innovation, and technology diffusion are on a continuum, and
Other U.S. Public and Private Innovation Surveys
there is still debate regarding their respective space on that
continuum. As NCSES develops surveys, new datasets, and BRDIS is not the only source of innovation-related data
new indicators of innovation activities, it will be important in the United States. Other agencies collecting information
to attempt to establish rigorous standards for defining these on innovation include the Bureau of Labor Statistics (BLS)
terms. Such standards will have implications for innovation in the U.S. Department of Labor, the Census Bureau, the
surveys internationally and for the comparability of data pro- Bureau of Economic Analysis (BEA), the National Institute
duced by those surveys. NCSES’s role in the working group of Standards and Technology (NIST) in the U.S. Depart-
of OECD’s National Experts on Science and Technology ment of Commerce, and the Economic Research Service in
Indicators (NESTI) gives the agency a good opportunity to the U.S. Department of Agriculture. Indeed, if the subject
contribute to the development of more precise standardized is broadened to STI, then at least six agencies collect these
definitions and their implementation through surveys such data, including the National Center for Education Statistics,
as the CIS and BRDIS when the Oslo Manual on innovation which collects data on STEM education.
(OECD-Eurostat, 2005) is next revised. The surveys administered by these agencies contain a
In addition to the lack of standard definitions, other factors wide range of useful global, national, state, and local indica-
can limit the comparability of U.S. and European data on the tors. However, most rely on policy indicators or indicators of
innovativeness of firms (see the discussion of international inputs to innovation activity; they contain very few measures
comparability later in this chapter in the section on improve-
ments to BRDIS). These factors include the use of different 10This is a reminder that innovation is difficult to measure, and inter-
survey frames, of different size cutoffs (BRDIS uses busi- national comparability is difficult to achieve. Not all business strategy is
nesses with 5 or more employees, while other countries use “innovation.” Improved business value and other success indicators can
businesses with 10 or 20 more employees), the impact of a come from various quarters. Can companies distinguish innovation from,
lengthy R&D and innovation survey, sampling errors, and say, “continuous improvement” or other noninnovation business enhancers?
How can innovation and its impact on a business be measured? These are
weighting issues. NCSES and OECD are actively collecting basic questions that the cognitive testing project is considering.
MEASURING INNOVATION 49
TABLE 4-2 Websites for Innovation Surveys from Regions Around the World
Region Countries Website
European Union (EU) 27 Austria, Belgium, Bulgaria, Cyprus, Czech Republic, http://ec.europa.eu/enterprise/policies/innovation/policy/
Denmark, Estonia, Finland, France, Germany, Greece, innovation-scoreboard/index_en.htm
Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg,
Malta, Netherlands, Poland, Portugal, Romania,
Slovenia, Slovakia, Spain, Sweden, UK
Type of Innovation 2006-2008 (%) 2007-2009 (%) 2008-2010 (%) 2009-2011 (%)
Goods 4.6 5.8 5.5 5.4
Services 6.9 9.1 7.8 6.8
Manufacturing/production methods 3.6 4.4 4.1 3.8
Logistics/delivery/distribution methods 3.1 4.1 3.7 3.2
Support activities 6.9 9.3 8.2 7.2
NOTE: In the 2008, 2009, 2010, and 2011 BRDIS, companies were asked to identify innovations introduced in 2006-2008, 2007-2009, 2008-2010, and
2009-2011, respectively.
SOURCE: Tabulations from the 2008-2011 BRDIS.
of innovative outputs. This gap illustrates the importance of First, surveys are time-consuming and expensive for the
BRDIS, and it also suggests that these agencies, to the extent companies asked to complete them, as well as for those that
possible, should be working together (see Recommendation must compile, validate, and publish the results. The quality
8-2 in Chapter 8). of the survey depends not only on the number of responses,
but also on the amount of time and level of personal atten-
tion paid to them by respondents, which surely vary across
Limitations of Innovation Surveys
companies. Surveys of individuals, such as those relating to
The panel strongly supports the increased use of innova- the career paths of scientists and engineers discussed later in
tion surveys as a way to measure the output of the innovation this report, are becoming increasingly problematic because
system. Later in this chapter, new, nontraditional methodolo- of declining response rates.11 NCSES also experiences
gies for measuring innovation are discussed. In this section,
11See
however, the panel underscores the inherent limitations of the National Research Council (2013a) report on survey nonre-
any survey of innovation inputs and outputs. sponse. The report finds that “for many household surveys in the United
States, response rates have been steadily declining for at least the past two
decades. A similar decline in survey response can be observed in all wealthy
delays in obtaining data on firms, as is clear from the delays search activity will not show up as an increase in consumer
in publishing statistics from BRDIS. expenditures (Mandel, 2012).
A second limitation of innovation surveys is that the Sixth, although economists have long viewed innovation
nature of the economy is rapidly changing, largely as a result as unfailingly positive, extending the so-called “production-
of innovation itself. As a consequence, new and interesting possibility” frontier, recent events suggest that this is not
subjects (such as outsourcing of R&D or the rise of “big always the case. Various (but not all) financial innovations,
data”)12 on which survey questions and answers would be for example, had perverse effects that contributed to the
desirable are constantly coming to the attention of policy financial crisis of 2007-2008, which in turn led to the Great
makers and researchers. Yet because of the many lags built Recession (see, e.g., Litan, 2010). Likewise, while innova-
into the survey process—from survey design, to survey tions and communication that have facilitated the growth of
deployment, to compilation and validation of results, some- trade can make some countries better off, some countries or
times across multiple agencies—results may be stale or of portions of their population (unskilled workers in particular)
limited use once they are published. Timing also is impor- may be made worse off in a globalized economy (see, e.g.,
tant because rates of dissemination of new knowledge and Samuelson, 1962).
products are important variables of interest. Furthermore, in Finally, the virtue of innovation surveys is that they
the current budgetary environment, it will be difficult if not provide an overall picture of firms’ innovation activity.
impossible for NCSES to mount new surveys in the future However, experience suggests that the subjective nature of
without cutting back on those it already undertakes. survey responses poses some difficulties for interpretation.
Third, large, multinational companies that report R&D In particular, the percentage of self-reported innovative firms
expenditures report where they conduct the R&D (and thus varies across countries in unexpected ways. According to the
spend money on salaries, equipment, and the like). They 2009 BRDIS, for example, only 6.4 percent of U.S. firms
rarely if at all, however, report to the public and government had introduced new or significantly improved goods in the
statistical agencies the impacts of their R&D on their opera- previous 3 years, while only 10.3 percent reported new or
tions, and specifically the extent to which the location of significantly improved services. By contrast, the percentage
the R&D leads (or does not lead) to the creation of jobs and of German firms reporting themselves as innovative in the
income in the United States. period 2005-2007 was far higher, at 79.9 percent.13 In part,
Fourth, the traditional classification of economic transac- this large and surprising disparity may reflect differences in
tions into goods and services can miss important innovations survey methodology and questions. Even within the Euro-
that do not fall easily into either of these categories. A good pean Union (EU), however, unexpected differences are seen,
example is the rise of data as a separate economic category or with Germany reporting far higher rates of innovation than
product. Innovations in constructing and analyzing databases the Netherlands (44.9 percent) and the United Kingdom
are becoming increasingly important in an ever-expanding (45.6 percent).
range of industries—not only Internet search, but also retail- The panel does not have recommendations for NCSES
ing and manufacturing and now even health care. These inno- for responding to each of these challenges. Rather, the chal-
vations are important, but their economic value is difficult to lenges are described here to give a sense of how changes in
measure and categorize. the economy lead to difficulties in measuring innovation and
Fifth, a more general challenge is that some innovations— its impact. The panel does have some thoughts about general
the precise share is unknown—do not result in increased processes NCSES can follow to stay abreast of these changes
output, at least as conventionally measured. Accordingly, and meet these challenges, which are outlined later in this
“production functions” that relate R&D to increased output and subsequent chapters of the report.
may not be capturing well the real economic impact (or spill-
over effects) of some innovations. For example, the activity
IMPROVEMENTS TO BRDIS
of someone in Germany downloading a free application writ-
ten in the United States will not show up in either the U.S. This section examines improvements that could be made
trade or output statistics. Similarly, an increase in Internet to BRDIS in five areas: (1) international comparability; (2)
deeper information on innovations; (3) extensions to cover
organizational and marketing innovations, unmarketed
innovations, and a broader array of inputs to innovation;
countries” (National Research Council, 2013a, p. 1). The report also docu- (4) improvements to the presentation of information; and
ments declining response rates for a variety of surveys, including panel and
longitudinal surveys (pp. 14-30).
(5) better linkages between BRDIS data and other datasets.
12The term “big data” technically refers to a collection of data so large
and complex that it is difficult to store in one place and difficult to manage.
Recently, however—and in this report—the term has come to refer to data
that are gathered from sources other than surveys, such as administrative
records, websites, genomics, and geographic sensors, to name a few. Statisti- 13The U.S. statistic was calculated from the NSF summary sheet on
cally, the term and its characteristics are still being codified. BRDIS. The statistics for German firms are found in Eurostat (2010).
MEASURING INNOVATION 51
International Comparability tant innovation, and experts then classified these examples
according to whether they met the requirements for an inno-
One impediment to understanding and assessing innova-
vation. The study results provide valuable information on
tion in the United States is the lack of comparability between
how respondents perceive innovations and on what innova-
U.S. STI indicators and those developed by other countries.
tion indicators represent. They also indicate which types of
Comparability of BRDIS and CIS data requires sur-
innovations are deemed most important for business perfor-
veying similar populations of firms in comparable size
mance; in many cases, examples of new organizational meth-
categories and using similar techniques to derive estimates
ods and new marketing concepts and strategies were cited.
of innovation from the raw data collected. Also needed for
NCSES might want to consider including an open ques-
comparability are statistics using the same set of industries
tion in BRDIS on the most important innovation or innova-
typically used in statistics for other countries. An example
tions produced by the firm over the previous 3 years. Alter-
is the core set of industries used by Eurostat for comparison
natively, NCSES might want to consider commissioning a
of innovation statistics among EU countries, including min-
study similar to that of Arundel and colleagues (2010) on a
ing; manufacturing; and selected service industries, such
subset of responses to BRDIS to determine what firms are
as wholesale trade, transport services, financial services,
measuring as innovation. It would be useful to have firms
information technology (IT) services, R&D services, and
of different sizes, different sectors, and different geographic
business services. These data could be used to compile a
locations represented in such a study.
simple indicator of the share of product-process innovative
firms, defined as firms that have implemented a product or
process innovation. Extensions to BRDIS
There are three important ways in which BRDIS could be
Deeper Information on Innovations extended to make it more useful for policy purposes. First,
the innovation questions could be broadened to include orga-
Even with more comparable statistics on innovation, it
nizational and marketing innovations. Second, it might be
still will not be clear to users that firms are representing the
appropriate for policy makers if the survey included the cat-
same or similar things when they report product or process
egory of “unmarketed” innovations, reflecting the reality that
innovations. The BRDIS questions do not give enough infor-
an innovation may be ready to be marketed but be held back
mation to provide a full understanding of what the resulting
by such factors as regulation, financial constraints, or the
data and statistics mean. For example, users have no inde-
market power of incumbents. Third, the survey could track
pendent measure of whether the innovations of firms that
a broader array of inputs to innovation, including non-R&D
innovate but do not conduct R&D are more or less important
investments such as design, market research, and databases.
than those of firms that conduct R&D. Users would have
more confidence in and understanding of the BRDIS innova-
tion measures if they knew that knowledge input measures Organizational and Marketing Innovations
were correlated with actual performance. Users would have
The communications sector, broadly defined to include
even more confidence in the measures if they knew what
communications-related hardware and software as well as
firms meant by innovation—how closely company reporting
telecommunications providers, has clearly been one of the
on the survey matched NCSES’s definition of innovation.
most innovative sectors of the economy in recent years.
Without greater detail on specific innovations, moreover,
Within this sector, smartphones are a prominent example
the surveys paint an exceedingly broad picture of innova-
of product innovation, while increases in mobile broadband
tion. Knowing, for example, that 60 percent of a country’s
speed exemplify process innovation. Other recent innova-
firms have introduced some type of innovation does not help
tions in the communications sector, however, do not fit so
in understanding why and how innovation happened, what
neatly into the product and process categories. These innova-
impacts it has on the economy, and how it can be encouraged.
tions include social media, such as Twitter, Facebook, and
Indicators should provide not only a level but also insight into
LinkedIn; search, exemplified by Google and Microsoft;
how that level was achieved. Microdata from innovation sur-
web marketing and advertising, exemplified by Google’s
veys connected with other data on a firm might help achieve
AdSense/AdWords technology (see Box 4-4); and the avail-
this goal, but this approach has as yet not been exploited.
ability of mobile applications through Apple’s App Store.
Most innovative firms introduce both product and process
To identify such successful and important innovations as
innovations, as well as organizational or marketing innova-
AdSense/AdWords, NCSES may need to broaden the scope
tions (discussed below), and the impacts of the innovations
of the innovation questions on BRDIS to include organiza-
are likely to depend on many other business decisions (see
tional and marketing innovations as identified by the Oslo
OECD, 2010; Stone et al., 2008).
Manual (see Box 4-5). In addition, NCSES might consider
In a recent study conducted in Australia (Arundel et al.,
introducing a question about significant new data algorithms
2010), respondents were asked to describe their most impor-
as a category of innovation. Such a question would require
14The panel notes that some regulations achieve societal goals that SOURCE: The 2012 Community Innovation Survey (Eurostat, 2012).
increase social well-being and that advancing innovation clearly is not the
only means of advancing a society’s goals.
MEASURING INNOVATION 53
potential innovations. For example, if companies lack access improved products or processes (excluding expendi-
to important sales channels that are controlled by incum- tures on equipment for R&D);
bents, then they may not find it worthwhile to commercialize • acquisition of existing know-how, copyrighted
a new product even if that product would be economically works, patented and non-patented inventions, etc.
viable in a more competitive market. from other enterprises or organisations for the devel-
Finally, in some cases, a potential innovation may not be opment of new or significantly improved products
introduced to market because of the need for other, comple- and processes;
mentary innovations. A classic example is Chemcor, an • in-house or contracted out training for your personnel
ultrahard glass invented in the 1960s by Corning. Because specifically for the development and/or introduc-
of the cost of production, Chemcor did not find a place in tion of new or significantly improved products and
the market until 2006. Renamed Gorilla Glass, it became the processes;
product of choice for cell phone screens. • in-house or contracted out activities for the market
The concept of unmarketed innovations is implicit in an introduction of your new or significantly improved
existing question on the CIS: goods or services, including market research and
launch advertising;
During the three years 2010 to 2012, did your enterprise have • in-house or contracted out activities to design or alter
any innovation activities that did not result in a product or the shape or appearance of goods or services; and
process innovation because the activities were • other in-house or contracted out activities to imple-
ment new or significantly improved products and
• abandoned or suspended before completion processes such as feasibility studies, testing, tooling
• still ongoing at the end of 2012
up, industrial engineering, etc.” (Eurostat, 2012,
p. 6).15
The panel believes NCSES should consider adding a
similar question to BRDIS. NCSES might also consider While having reliable data on these innovation-related
asking respondents to rank the main reasons why the outputs expenditures would be useful, it may be difficult for com-
of their innovation activities have not yet been marketed panies to report these expenditures if they do not already
or implemented, including not yet ready, lack of funding, record them in their accounts. Interviews and testing with
lack of sufficient skilled labor, the need to meet regulatory companies would help in discerning which types of expen-
requirements, or blocked from market by competitors. The ditures can be reported reliably. Expenditures on training
panel acknowledges that such questions may be difficult for and design may be among the most feasible to measure and
respondents to answer, although such a question on factors are important for non-R&D innovation activities. Data on
hampering product and process innovation has been asked innovation-related expenditures may also provide useful
in the CIS for some years (Eurostat, 2012). If such questions input for the development of statistics on knowledge-based
were added to BRDIS, then the insights they would yield into capital and “intangible assets” (discussed in Chapter 5).
bottlenecks for innovation could be quite useful.
MEASURING INNOVATION 55
Linkages Between BRDIS Data and Other Datasets of the United States on an aspect of the country’s innovation
capacity.
The data gathered in BRDIS could be used to begin
NCSES has a unique set of data in BRDIS, which, if com-
developing statistics on high-growth firms and “gazelles.”
bined with other datasets, could be instrumental in answering
The Manual on Business Demography Statistics (OECD-
these and other important questions. Integrating data on firm
Eurostat, 2008, Chapter 8, p. 61) defines high-growth enter-
dynamics (and the related employment effects) would take
prises as “all enterprises with average annualised growth
time and resources. During his presentation at the workshop,
greater than 20% per annum, over a three year period. . . .
Haltiwanger described three Census Bureau datasets that,
A size threshold has been suggested as 10 employees at the
together with BRDIS data, would allow NCSES to develop
beginning of the growth period.” Gazelles are the subset of
indicators of business dynamics:
high-growth enterprises that are up to 5 years old (OECD-
Eurostat, 2008, Chapter 8, p. 63). These thresholds are arbi-
• Longitudinal Business Database—tracks all estab-
trary and based only on convention. NCSES could conduct
lishments and firms with at least one employee,
its own sensitivity analysis to fine-tune the definitions of
including start-ups, from 1976 to the present;
high-growth firms and gazelles.17
• Integrated Longitudinal Business Database—tracks
During the panel’s July 2011 workshop, several speak-
all nonemployer firms and integrated-with-employer
ers18 mentioned the importance of tracking trends in the
firms from 1994 to the present; and
sustainability of jobs in these types of firms during economic
• Longitudinal Employer-Household Dynamics—
downturns (even if total employment is small). It would also
tracks longitudinally all employer-employee matches
be useful to have firm data by age classes to determine over
and transitions (hires, separations, job creation, and
time whether high-growth firms or gazelles in particular
job destruction) from 1990 to the present.
have a higher incidence of innovation activity relative to
other firms. In his presentation at the July 2011 workshop,
Questions from the Census Bureau’s 2007 and 2012 Eco-
Hollanders showed that high-growth firms are significantly
nomic Census, Company Organization Survey, and Man-
more innovative than other firms in his dataset. The connec-
agement and Organizational Practices Survey can also yield
tion between high-growth firms and innovation is complex,
useful information on R&D and other innovation activities
and these data would help researchers better understand it.
for establishments. In addition, infrastructure datasets can
Statistics on high-growth firms and gazelles could also be
track relationships between start-up and young high-growth
used to answer the question of whether these types of firms
firms and large, mature firms, and can be linked further to
drive economic and job growth. A simple table could com-
patent and citation data. Important as well is to link data on
pare the economic characteristics of high-growth and other
firm dynamics to those on innovation outputs, such as patent
firms that are and are not innovative, ideally over time.
and citation data.
At the panel’s September 2011 meeting in Washington,
Haltiwanger proposed that indicators track firm dynam-
DC, representatives from BLS, the U.S. Census Bureau,
ics by geography, industry, business size, and business age.
and BEA mentioned that linking certain datasets among
Hollanders noted that European countries and other OECD
them would yield reasonable numbers on gazelles. A table
members are continuing to fine-tune their measures of firm
with these numbers could be added to the Science and
dynamics. NCSES’s indicators on this dimension could
Engineering Indicators or become the foundation of an
further the international comparability of its STI indicators.
InfoBrief. The following indicators could be produced using
Building the foundations for indicators of firm dynamics
BRDIS and other data on high-growth firms and gazelles:
using BRDIS and other datasets would give NCSES a pro-
number of high-growth enterprises as a percentage of the
ductive platform for developing several STI indicators that
total population of active enterprises with at least n-number
are policy relevant.
of employees, and number of gazelles as a percentage of
Clearly, developing publishable statistics on high-growth
all active enterprises with at least n-number of employees
firms and gazelles is a multistage task requiring data acquisi-
that were born 4 or 5 years ago. These indicators would be
tion and linkage in addition to use of the data available from
comparable to those produced in several other countries, thus
BRDIS. A good first step would be for NCSES to explore
increasing users’ understanding of the comparative position
linking its BRDIS data with data on firm dynamics from
BLS.
17Petersen and Ahmad (2007) present a technique for conducting this
type of analysis in OECD (2007).
RECOMMENDATION 4-4: The National Center for
18Howard Alper, University of Ottawa; Robert Atkinson, Information Science and Engineering Statistics (NCSES) should
Technology and Innovation Foundation; John Haltiwanger, University begin a project to match its Business Research and
of Maryland; Hugo Hollanders, United Nations University’s Maastricht Development and Innovation Survey data to data
Economic and Social Research Institute on Innovation and Technology from ongoing surveys at the U.S. Census Bureau
(UNU-MERIT); and Brian MacAulay, National Endowment for Science,
Technology, and the Arts.
and the Bureau of Labor Statistics. It should use
the resulting data linkages to develop measures of ing activities,19 and (5) trademark databases.20 This section
activities by high-growth firms, births and deaths of examines the advantages and disadvantages of business prac-
businesses linked to innovation outputs, and other tice data from the perspective of NCSES. Further discussion
indicators of firm dynamics, all of which should be of the use of these types of data for developing STI indicators
tabulated by geographic and industry sector and by appears in Chapter 7 of this report.
business size and business age to facilitate compara-
tive analyses. NCSES should conduct a sensitivity
Conceptual Background
analysis to fine-tune meaningful age categories for
high-growth firms. The use of business practice data as a supplement to
traditional surveys is under active consideration in statisti-
cal agencies. Robert Groves, then director of the Census
USE OF NONTRADITIONAL METHODOLOGIES
Bureau, recently delineated the key issues entailed in using
Traditionally, NCSES and its predecessors have used sur- these data (see Box 4-7), noting: “Combined, the ‘big data’
veys to trace the inputs and outputs of the innovation system. and the benchmark survey data can produce better statistics.”
As noted earlier, however, executing a survey is an expensive The advantages of business practice data include
and time-consuming process, requiring writing and testing
questions, identifying the universe of potential respondents, • Timeliness—Collecting and analyzing survey data
collecting the data, cleaning and validating the data, analyz- is a lengthy process. Because most business practice
ing the data, and then finally publishing the results. Thus, for data today are digital, they can be available for analy-
example, NCSES did not publish the topline R&D spending sis in near real time.
results from the 2009 BRDIS until March 2012. • Detail—Survey data offer limited detail by industry
Another issue with government surveys is that those and geographic location. Business practice data can
being surveyed may not respond, and when they do, they are be used to track innovation activity by detailed indus-
almost always guaranteed confidentiality. As a result, some try or subnational area.
or even many results must be withheld to avoid disclosing, • Flexibility—Survey methods require making some
even indirectly, the responses of individual companies. In the assumptions about the nature of innovation many
basic table showing R&D spending by industry and source years in advance. The ability to adjust the measuring
of funding from the 2010 and 2011 BRDIS (Table 2 in Info- tool easily is helpful, particularly when new catego-
Brief 13-335), for example, 37 and 36 (respectively) of the ries of innovation emerge.
154 cells are suppressed. The significance of this problem
grows when more detailed industries or subnational areas At the same time, business practice data have disadvantages
are considered. Take, for example, the interesting ques- that make them an imperfect substitute for conventional
tion of whether the location of R&D conducted by the IT surveys. These disadvantages include
industry is becoming more or less concentrated in Silicon
Valley. If BRDIS were to ask companies to apportion their • “Institutional drift”—Business practice data are
research spending by metro area, the data for most metro generated by normal business activity, but because
areas across the country would likely have to be suppressed patterns of business activity change over time, inter-
because information on a specific firm might be identified in pretations of business practice data are not necessar-
cases involving a small number of firms in a given sector or ily stable.
region. It might even be difficult to obtain useful data at the • Difficulty with cross-industry/cross-country
state level for the same reason. comparisons—Different industries and countries
Until fairly recently, there was no good alternative to may have very different business practices.
surveys for collecting data on innovation inputs and outputs.
Increasingly, however, businesses and individuals are gener-
ating detailed electronic data in the normal course of their
economic activity. Such data are available either in a firm’s 19“Altmetrics are alternative measures that can supplement citation counts
administrative records, publicly on the web, or from third and journal impact factors as measures of the impact of scholarly commu-
parties that collect them in the normal course of economic nications. Such measures are generally derived from online activity such as
activity, and can be obtained in digital form from a given mentions, downloads, tweets, blog posts, Facebook “likes,” bookmarking
and other similar evidence of attention” (Travis, 2013).
firm. Such nontraditional data are referred to here as “busi- 20Note that not all business practice data fall into the category commonly
ness practice” data. Examples of these data include several referred to as “big data.” For example, administrative records are included
datasets derived from the Internet: (1) reports on innovations, in the panel’s definition of business practice data but are not considered
(2) help-wanted ads, (3) journal articles on new products and “big data.” Also note that the term “big data” had not been formally codi-
services, (4) altmetric measures of scientific and engineer- fied by statistical agencies at the time this report was being written. Lastly,
“big data” is not a panacea (see Boyd and Crawford [2011] for important
caveats).
MEASURING INNOVATION 57
RECOMMENDATION 4-5: The National Center • model name, trade name, or trademark;
for Science and Engineering Statistics should make • name and description of the innovation;
greater use of business practice data to track research
21Nonsurvey methods for extracting data for the development of STI
and development spending and innovation-related
indicators are discussed in greater detail in Chapter 7.