Measuring Firm Innovation

Download as pdf or txt
Download as pdf or txt
You are on page 1of 19

This PDF is available at http://nap.nationalacademies.

org/18606

Capturing Change in Science, Technology,


and Innovation: Improving Indicators to
Inform Policy (2014)

DETAILS
274 pages | 8.5 x 11 | HARDBACK
ISBN 978-0-309-38717-0 | DOI 10.17226/18606

CONTRIBUTORS
Robert E. Litan, Andrew W. Wyckoff, and Kaye Husbands Fealing, Editors; Panel on
Developing Science, Technology, and Innovation Indicators for the Future;
BUY THIS BOOK Committee on National Statistics; Division of Behavioral and Social Sciences
and Education; Board on Science, Technology, and Economic Policy; Policy and
Global Affairs; National Research Council

FIND RELATED TITLES SUGGESTED CITATION


National Research Council. 2014. Capturing Change in Science, Technology, and
Innovation: Improving Indicators to Inform Policy. Washington, DC: The
National Academies Press. https://doi.org/10.17226/18606.

Visit the National Academies Press at nap.edu and login or register to get:
– Access to free PDF downloads of thousands of publications
– 10% off the price of print publications
– Email or social media notifications of new titles related to your interests
– Special offers and discounts

All downloadable National Academies titles are free to be used for personal and/or non-commercial
academic use. Users may also freely post links to our titles on this website; non-commercial academic
users are encouraged to link to the version on this website rather than distribute a downloaded PDF
to ensure that all users are accessing the latest authoritative version of the work. All other uses require
written permission. (Request Permission)

This PDF is protected by copyright and owned by the National Academy of Sciences; unless otherwise
indicated, the National Academy of Sciences retains copyright to all materials in this PDF with all rights
reserved.
Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

Measuring Innovation

This chapter considers the measurement of innovation— essential that such measures be used in concert with in-
in particular, the outputs of the country’s innovation sys- depth analysis of economic and institutional environments
tem.1 It presents definitions of innovation, explains the and understanding of behavioral responses to changes in
importance of measuring innovation and the relevance of those environments. An innovation may reach forward into
innovation measures for policy decisions, and examines varied markets and backward to human capital and at times
the role of innovation surveys and their limitations. The research and development (R&D) inputs. The discussion in
chapter then reviews improvements to innovation surveys, this chapter must therefore be viewed in the context of an
in particular the Business Research and Development and indicators program that is analytically strong.
Innovation Survey (BRDIS), including improving inter-
national comparability, gathering deeper information on
DEFINITIONS
innovations, extending BRDIS to include such items as
organizational and marketing as well as “unmarketed” Schumpeter (1934, p. 66) provided a definition of “inno-
innovations and to track a broader array of inputs to inno- vation” early in the 20th century. He defined product innova-
vation, improving the presentation of information, and tion as “the introduction of a new good . . . or a new quality
improving linkages between the BRDIS data and other of a good” and process innovation as “the introduction of a
datasets. Finally, the chapter turns to the use of nontradi- new method of production . . . or a new way of handling a
tional methodologies to track innovation, such as the use commodity commercially.” This definition influenced early
of “business practice” data. attempts to measure the activity of innovation through case
In accordance with the framework used in this study studies and surveys. Other definitions offered more recently
(Chapter 2), the panel’s recommendations for the National are presented in Box 4-1.
Center for Science and Engineering Statistics (NCSES) Knowledge gained from a decade of experience with
regarding the measurement of innovation are driven by key experimental surveys in Canada, Germany, the Nordic
policy questions and by specific questions raised by users of countries, and the United States was codified by OECD in
science, technology, and innovation (STI) indicators, as well the first Oslo Manual (OECD, 1992), which dealt with inno-
as by the need for internationally comparable measures. The vation in manufacturing products and processes. Following
panel acknowledges that the innovation measures presented more measurement experience, the scope of the manual was
in this chapter answer policy questions only in part. It is extended to the nonagricultural economy (OECD-Eurostat,
1997). This broadened scope gave rise to a better understand-
1Edquist (2005, p. 182) defines the “system of innovation” as “all im- ing of the measurement of innovation in service industries
portant economic, social, political, organizational, institutional, and other and expansion of the definition of innovation to include not
factors that influence the development, diffusion, and use of innovations.”
Lundvall (1992, p. 2) states that a system of innovation is “constituted by
only product and process innovations but also organizational
elements and relationships which interact in the production, diffusion and change and the development of new or the extension of exist-
use of new, and economically useful, knowledge.” Lundvall (2007, p. 106) ing markets. These additional components of the definition
extends this definition by focusing on the learning processes, stating: “The were also considered by Schumpeter.
analysis of innovation systems may be seen as an analysis of how knowl- For measurement purposes, the Oslo Manual (OECD-
edge evolves through processes of learning and innovation.” Other noted
studies on national innovation systems include Freeman (1995); Furman et
Eurostat, 2005, p. 46) defines innovation as follows:
al. (2002); Lundvall (1988); Martin and Johnston (1999); Mowery (1992);
Muller and Zenker (2001); and Nelson (1993).

41

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

42 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

BOX 4-1
Definitions of Innovation

The following definitions of innovation vary, but the common thread is the extraction of economic value from novel activities (Innovation Vital Signs
Project 2007):

Innovation is “the commercial or industrial application of something new—a new product, process or method of production; a new market or sources
of supply; a new form of commercial business or financial organization.”
Schumpeter 1983

Innovation is the “intersection of invention and insight, leading to the creation of social and economic value.”
Council on Competitiveness 2005

Innovation covers a wide range of activities to improve firm performance, including the implementation of a new or significantly improved product,
service, distribution process, manufacturing process, marketing method or organization method.
European Commission 2004

Innovation—the blend of invention, insight and entrepreneurship that launches growth industries, generates new value and creates high value jobs.
Business Council of New York State 2006

The design, invention, development and/or implementation of new or altered products, services, processes, systems, organization models for the
purpose of creating new value for customers and financial returns for the firm.
Advisory Committee on Measuring Innovation in 21st Century Economy, Department of Commerce 2008

An innovation is the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new
organizational method in business practices, workplace organization or external relations. Innovation activities are all scientific, technological, orga-
nizational, financial and commercial steps which actually, or are intended to, lead to the implementation of innovations.
OECD-Eurostat 2005

Innovation success is the degree to which value is created for customers through enterprises that transform new knowledge and technologies into
profitable products and services for national and global markets. A high rate of innovation in turn contributes to more market creation, economic
growth, job creation, wealth and a higher standard of living.
Innovation Vital Signs Project 2007

SOURCE: Aizcorbe et al. (2009).

. . . the implementation of a new or significantly improved of collecting information on “unmarketed” innovations to


product (good or service), or process, a new marketing address important policy questions.2
method, or a new organizational method in business prac- By contrast, the Oslo Manual (OECD-Eurostat, 2005, p.
tices, workplace organization or external relations. 18) identifies such inputs as R&D, capital expenditures, and
training as “innovation activity”:
Innovations can be distinguished from inventions by the
criterion that innovations are implemented in the market- . . . all scientific, technological, organisational, financial and
place. By definition, therefore, innovation is a new product commercial steps which actually, or are intended to, lead to
or idea that has been commercialized. This does not mean the implementation of innovations. Some innovation activi-
that an innovation is necessarily widely distributed or dif- ties are themselves innovative; others are not novel activities
fused in a market. It does mean that neither a new product but are necessary for the implementation of innovations.
that is not marketed nor a new process that does not help Innovation activities also include R&D that is not directly
get a product to market is considered an innovation. Note related to the development of a specific innovation.
that later in this chapter, the panel considers the possibility
2The European Community Innovation Survey (CIS) has been gathering

such information, which has been shown to be policy relevant. For CIS 2012,
the relevant question is 4.1 (Eurostat, 2012).

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 43

WHY MEASURE INNOVATION? intellectual property rights. Also, the propensity to patent is
highly industry dependent.
The underlying analytical framework used in this study
Similarly, input measures such as R&D spending cannot
posits that new and improved products and processes, new
adequately substitute for measures of innovation output,
organizational methods, and new marketing concepts and
especially if policy makers are concerned about government
strategies are key measures of the output of a country’s
funds being used efficiently. Moreover, recent evidence
innovation system. Innovation affects both economic per-
shows that most firms that report innovations do not report
formance measures, such as productivity growth, profits,
R&D spending. For example, Table 4-1, drawn from the 2011
and job creation, and noneconomic variables, such as life
BRDIS results, shows that 79 percent of all firms with new or
expectancy and environmental response. Meanwhile, the
significantly improved products or processes did not report
rate of innovation responds to inputs such as R&D spend-
R&D spending. In other words, focusing only on firms that
ing; the availability of science, technology, engineering, and
report R&D spending raises the possibility of missing much
mathematics (STEM) labor; regulatory policies; and other
of the innovation taking place in the economy.4
variables.3
Many innovations developed and introduced by start-up
From this perspective, the panel strongly believes that
companies may not be associated with formal R&D, or R&D
NCSES needs to improve its ability to measure and track
may not be recorded as such because start-ups often have no
innovation. Improved measures of innovation are necessary
revenues against which to record expenses. More broadly,
to assess the impact of federal, state, and local innovation
advances in information and communication technologies
policies, such as the amount and direction of federal R&D
and the importance of innovations such as Google’s search
funding, support for STEM education at the graduate level,
service, Amazon’s e-commerce website, and Apple’s iTunes
and regulation of new products and services. In addition,
store signal how the nature of the innovation process has
having good measures of innovation output facilitates com-
changed, increasingly depending on investments in inno-
parison of the United States with other countries in a key area
vative assets other than R&D (e.g., data, organizational
that promotes economic growth. NCSES’s mandate in the
know-how) and with take-up primarily in the service sector.
America Creating Opportunities to Meaningfully Promote
Indeed, in fiscal year 2006, the year before the iPhone was
Excellence in Technology, Education, and Science (America
introduced, Apple reported spending less than 3 percent
COMPETES) Act (U.S. House of Representatives, 2010)
of its revenues on R&D—relatively low for a technology
includes the curation and dissemination of data on “United
company.5
States competitiveness in science, engineering, technology,
Finally, broad measures of economic performance, such
and research and development.” Innovation is an important
as productivity, do contain information about the country’s
element for such comparisons. Without improved direct indi-
innovation system, especially over longer periods such as
cators of innovation outputs, policy analysis will continue to
decades. However, productivity growth is a weak signal in
rely on imperfect indicators of innovation, such as number
the short run because it is affected by external factors such
of patents granted; inputs to innovative activities, such as
as the business cycle and by external factors such as the
R&D spending and number of STEM workers; and broad
availability of finance.
performance measures for the economy, such as productivity.
The panel acknowledges that developing useful measures
A voluminous literature addresses the historical link
of innovation output is far more difficult than measuring
between patents and innovation, and the topic remains con-
inputs or patents, for example. The space of economically
troversial today. However, there is little doubt that the more
significant innovations is both diverse and expanding. Inno-
than doubling of patent grants over the past 20 years—from
96,511 in 1991 to 224,505 in 2011—did not reflect an equally 4Sectors in which more than 50 percent of firms indicated that they were
sharp rise in the volume of innovation. As recent litigation
engaged in either product or process innovation between 2009 and 2011
has shown, multiple patents have become a bargaining chip were manufacturing—petroleum and coal products; chemicals (particularly
rather than an accurate measure of innovation output. More- basic chemicals; soap cleaning compound, and toilet preparation; paint,
over, a recent NCSES InfoBrief suggests that patents are coating, adhesive, and other chemicals); machinery (particularly agricul-
less important to companies than other forms of intellectual tural implement; engine, turbine, and power transmission equipment);
computer and electronic products (particularly communications equipment;
property protection (Jankowski, 2012). Based on data from
semiconductor and other electronic components; navigational, measuring,
the 2008 BRDIS, firms across virtually every industry are electromedical, and control instruments; other measuring and controlling
much more likely to cite trademarks, copyrights, and trade devices); and transportation equipment (particularly guided missile, space
secrets than either design or utility patents as important for vehicle, and related parts); or nonmanufacturing—information (particularly
software publishers). See Appendixes H-K for specific details.
5Apple’s R&D/sales ratio has historically been relatively low compared
3One prominent model is presented by Crépon and colleagues (1998). with companies such as Google, Microsoft, Intel, and Samsung. For ex-
This three-equation model has the following structure: (1) RD = f (Xr); ample, Apple’s R&D/sales ratio was at 2.27 percent in fiscal year 2012,
(2) Innovation = g(RD, Xi); and (3) Productivity/other economic output approximately one-third of Samsung’s R&D/sales ratio. However, Apple
= h(Innovation, Xp), where X’s are vectors of other factors relevant to the may have benefited from R&D spending by suppliers such as Samsung
output (see also Mairesse and Mohnen, 2010; OECD, 2009). and Intel.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

44 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

TABLE 4-1 Research and Development (R&D) Firms Are More Likely to Innovate, but Most Innovating Firms Do Not Do
R&D
% of Firms with New or Significantly Improved Products or Processes % of Total Firms in Scope*
R&D Status of Firm Doing R&D Not Doing R&D Total
Firms Doing R&D 64 N.A. 21 5
Firms Not Doing R&D N.A. 12 79 95

NOTE: A firm that does R&D may produce many innovations, while a firm without R&D may produce only one innovation. However, BRDIS does not
report on the number of innovations per firm.
*Approximately 1.2 million firms on a weighted basis are in scope, that is, reported to the survey.

SOURCE: Calculated by the panel from the 2011 BRDIS, Table 49; see Appendix K. For more information on innovation statistics from the 2011 cycle
of BRDIS, see Business R&D and Innovation: 2011, which will be published in 2014. For previous cycles, the full set of detailed statistical tables is available
in Business R&D and Innovation: 2008-2010 (NSF 13-332) at http://www.nsf.gov/statistics/nsf13332/start.cfm [April 2014]. Innovation statistics from the
2008 BRDIS are found in Tables 33-36, the 2009 BRDIS innovation statistics are found in Tables 82-85, and the 2010 BRDIS innovation statistics are found
in Tables 126-129. Relative standard errors are available from NCSES staff upon request.

vation can include everything from the successful develop- such indicators could help address. This is not to say that the
ment and approval of gene therapy, to the development of the indicators suggested here are relevant only for government
special high-strength glass used for smartphone screens, to policy makers; many of them should be useful to university
the steady improvement of unmanned aerial vehicles used by administrators and business managers as well.
the military in Afghanistan, to logistical warehouse manage- As noted above, the value of a measure of innovation is
ment systems. Other significant contemporary innovations that it offers the possibility of a direct link between innova-
include Google’s AdSense/AdWords technology for targeted tion outcomes and relevant policy variables, many of which
online advertising and Apple’s invention of the App Store are innovation inputs. In particular, one or more innovation
concept, which enables small programming firms to sell and measure can make it possible to test the impact on innovation
get paid for lightweight mobile applications. Although some of such factors as R&D spending, the availability of STEM
may disagree, moreover, the creation of credit default swaps labor, regulation, market power, globalization of research,
in the 1990s was almost certainly an economically significant immigration policy, and tax policy (see Box 4-2). Note that
innovation of a type that should be considered by NCSES.6 these factors can have either a positive or a negative effect
Nevertheless, it is clear that as a national goal, policies on innovation, and it is essential for policy makers to know
that encourage bringing more innovations to market are use- which way the arrow points and under what conditions.
ful if they generate economic growth and jobs and improve Firm-level analysis also links investment in innovation to
the nation’s competitiveness. As discussed below, measures improved outcomes. Using the third Community Innovation
of innovation outputs are needed, and providing such mea- Survey (CIS3) for 16 European countries, Jaumotte and Pain
sures is clearly within the purview of NCSES to support (2005) found that the proportion of firms engaged in inno-
such policies. vation spending was closely correlated with the proportion
of successful innovators in a country. Their results suggest
RECOMMENDATION 4-1: The National Center for that a 1 percentage point increase in aggregate innovation
Science and Engineering Statistics should develop spending was associated with an increase of 0.85 percent-
additional indicators for measuring innovation age points in the probability of being a successful innovator
outcomes that would complement existing data on and an increase of 0.7 percentage points in the share of new
patents, inputs to innovation activities, and broader products in turnover.7
measures of economic performance. An analysis of innovative companies could shed light on
such key policy questions as the appropriate amount and
direction of federal R&D funding, limits on or encourage-
POLICY RELEVANCE OF INNOVATION MEASURES
ment of immigration of skilled workers, support for STEM
Information should not be collected in a vacuum. It is
good business practice for NCSES to focus on developing 7In this study, innovation spending included not only intramural R&D but

new indicators that are useful for informing policy decisions, also extramural R&D, other acquisitions of knowledge (such as rights to use
especially given the current fiscal environment. This section patents, licenses, and software), investment in capital goods, expenditures
on training, and spending necessary for the innovation to be placed on the
identifies some innovation-relevant policy questions that
market (for example, marketing or design expenditures). Non-R&D expen-
ditures were sometimes more important than those on R&D, and many of the
6According to one account, the concept of credit default swaps was countries with the highest proportion of successful innovators also had the
developed in 1994 to help Exxon fund its huge potential liability for the highest propensity to engage in non-R&D innovation spending—showing
Exxon Valdez disaster (see Tett, 2009). the importance of taking a broad view of innovation inputs.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 45

Institutes of Health having struggled over the past decade


with how to turn increases in scientific knowledge into mar-
BOX 4-2
Selected Innovation-Related ketable innovations that improve human health outcomes. In
Policy Questions this context, innovation measures could be helpful in two key
ways. First, they would quantify the perception that commer-
• What kinds of innovations create jobs and what kinds of jobs? cialization of biomedical research is slower than expected.
• What specific innovations have made the most important Second, they could help researchers offer insight into dif-
contributions to economic growth and over what time period? ferent possible explanations for the gap between knowledge
• Where should the government spend its R&D dollars? and the market, providing useful guidance for policy makers.
• Can government procurement policy be changed to encourage On the broadest level, measures of innovation can help
innovation? answer important policy-relevant questions such as those
• What new industries, occupations, and skill sets are being listed in Box 4-2. The list of such questions is easily much
created by recent innovations, and are current educational longer. However, current STI indicators are not fully capable
institutions equipping enough people with the skills needed of answering many of these questions.
to take maximum advantage of these changes?
• How important is formal R&D to the innovative process, as
opposed to organizational changes and continued on-the-job
THE ROLE OF INNOVATION SURVEYS
tinkering by employees and their managers? Currently, the most widely used tool for measuring
• Does the globalization of R&D increase or decrease the ability innovation is the innovation survey. Innovation surveys ask
to turn research into marketed innovations? firms about types of innovation in which they are engaged,
- Are U.S. firms conducting more or less of their research reasons for innovating (or not), collaboration and linkages
activities offshore and does it even matter where this among firms or public research organizations, and flows of
activity takes place? knowledge; they also collect quantitative data on sales of
- Is the United States innovating more or less than other product innovations and spending on a range of innovation
countries? activities beyond R&D. They are designed to collect direct
• Are small companies more likely to innovate than big com- information on the elements of the knowledge-to-production
panies? In which types of firms—small or large, young or system without asking about particular innovations or details
mature—do the most important innovations occur, and is this thereof. They provide fairly detailed guidance to respondents
pattern stable? about what is or is not an innovation, and allow for com-
• How do service firms innovate and create new knowledge? parison of levels of reported innovation across organizations/
• What are the sources of funding for innovation activities, countries and for analysis of the determinants of reported
including R&D? innovation and their link to economic output (Stone et al.,
• Are potential innovations not reaching market because of too 2008).
much regulation, taxes that are too high, or too much market
power in product markets?
• Are potential innovations being stalled because of insufficient
The Business Research and Development Innovation
funding, not enough skilled workers, or too many obstacles to
Survey (BRDIS)
skilled immigrants? In 2008, the National Science Foundation (NSF) launched
• Are innovations not being brought to market or implemented the BRDIS, which collects key information on innovation as
because of misguided patent policies? well as a wide range of other variables. Although the 2008
BRDIS was a pilot survey, it did yield some data on the
incidence of product and process innovation among firms by
sector (including services), size class, and whether respon-
dents reported R&D activity (Borousch, 2010).8 Questions
education at the graduate level, regulation of new products on innovation were augmented in the 2009 and 2010 ver-
and services, rules for university technology transfers, and sions of the survey, allowing for comparison of innovation
the appropriate role of R&D tax credits. Perhaps more impor- statistics across several countries.9 Box 4-3 has selected
tant, innovation measures should help identify bottlenecks in innovation questions from the 2011 version of BRDIS (U.S.
the innovation system—cases in which most of the neces-
sary inputs are present, but innovation apparently falls short 8The data are based on the 2008 BRDIS, which was launched in

of expectations. These are potentially high-return areas for January 2009. See National Science Foundation (2013a) for the BRDIS
policy, where relatively small policy changes could poten- questionnaire.
9It is widely known that the innovation statistics from BRDIS and the
tially lead to much better results.
CIS lack comparability; see an explanation in Hall (2011). Also see the
The most striking potential bottleneck is in biomedical discussion of the disparities between the U.S. and European innovation
research, with the pharmaceutical industry and the National statistics later in this chapter.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

46 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

BOX 4-3
Innovation Questions on the 2011 Business Research and
Development and Innovation Survey (BRDIS)

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 47

SOURCE: Available: http://www.nsf.gov/statistics/ [January 2014].

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

48 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

Department of Commerce, 2012), which contained sections evidence to assess the factors that may drive biases in inter-
on: the 2011 survey was in the field as of this writing (see national comparisons. (See Table 4-2 for a list of websites for
Box 4-3 for selected innovation questions from the 2011 innovation surveys from regions around the world.)
survey). The 2011 version of the BRDIS (U.S. Department Complete cognitive testing of the questions used on
of Commerce, 2012) contained sections on: innovation surveys is ongoing in the United States and
Europe. Nevertheless, the data are useful for preliminary
• innovation, reported by the nature of the innovation snapshots. OECD-NESTI has been conducting a cognitive
(new product or new process), and the percentage of testing research project since 2011, with participation by
sales coming from newly introduced products; representatives from the United States and other countries.
• R&D spending, reported by business code, country, The project is examining how businesses define and mea-
state, funder, and performer; sure innovation. This effort is motivated by the sizable gap
• characteristics of R&D spending, reported by type between estimates in the United States and Europe on the
(basic versus applied versus development) and area incidence of innovation among firms, with 2008 BRDIS data
of focus (e.g., biotechnology, nanotechnology); being used as the baseline for U.S. innovation statistics. The
• R&D employees, by type (scientists and engineers 2009 and 2010 BRDIS instruments included some systematic
versus technicians and technologists); and variations aimed at determining whether the way innovation
• use of intellectual property, including patents, and questions were being asked substantially influenced the
trademarks. incidence answers provided. During a discussion with panel
members, NCSES staff indicated that the conclusion to date
NCSES’s efforts to revise BRDIS are aimed at gathering is that there is no major difference in the statistics due to the
more information on innovation activities, going beyond phrasing of the questions. The weak result on innovation
simple “yes/no” questions on whether a firm introduced or changed little when the questions were moved to the front of
significantly improved goods, services, or processes. These the 2009 BRDIS (see Table 4-3). The relatively low reported
efforts also are aimed at enhancing comparability with key incidence of innovation in the United States compared with
questions in the CIS and ensuring that the questions on European nations remains an issue for further research by
innovation are answered by firms that do not conduct R&D. NCSES and its partners in OECD-NESTI.10 In addition, the
Comparability of the BRDIS and CIS data also depends on size dependence of the propensity to innovate may contribute
surveying similar populations of firms and on the techniques to the weak results, suggesting that the Table 4-3 should be
used to derive estimates from the data. produced for at least three employment cutoffs (see Recom-
As befits a cutting-edge survey, BRDIS is still evolving. mendation 4-2 later in this chapter).
It is important to recognize that such concepts as invention,
innovation, and technology diffusion are on a continuum, and
Other U.S. Public and Private Innovation Surveys
there is still debate regarding their respective space on that
continuum. As NCSES develops surveys, new datasets, and BRDIS is not the only source of innovation-related data
new indicators of innovation activities, it will be important in the United States. Other agencies collecting information
to attempt to establish rigorous standards for defining these on innovation include the Bureau of Labor Statistics (BLS)
terms. Such standards will have implications for innovation in the U.S. Department of Labor, the Census Bureau, the
surveys internationally and for the comparability of data pro- Bureau of Economic Analysis (BEA), the National Institute
duced by those surveys. NCSES’s role in the working group of Standards and Technology (NIST) in the U.S. Depart-
of OECD’s National Experts on Science and Technology ment of Commerce, and the Economic Research Service in
Indicators (NESTI) gives the agency a good opportunity to the U.S. Department of Agriculture. Indeed, if the subject
contribute to the development of more precise standardized is broadened to STI, then at least six agencies collect these
definitions and their implementation through surveys such data, including the National Center for Education Statistics,
as the CIS and BRDIS when the Oslo Manual on innovation which collects data on STEM education.
(OECD-Eurostat, 2005) is next revised. The surveys administered by these agencies contain a
In addition to the lack of standard definitions, other factors wide range of useful global, national, state, and local indica-
can limit the comparability of U.S. and European data on the tors. However, most rely on policy indicators or indicators of
innovativeness of firms (see the discussion of international inputs to innovation activity; they contain very few measures
comparability later in this chapter in the section on improve-
ments to BRDIS). These factors include the use of different 10This is a reminder that innovation is difficult to measure, and inter-
survey frames, of different size cutoffs (BRDIS uses busi- national comparability is difficult to achieve. Not all business strategy is
nesses with 5 or more employees, while other countries use “innovation.” Improved business value and other success indicators can
businesses with 10 or 20 more employees), the impact of a come from various quarters. Can companies distinguish innovation from,
lengthy R&D and innovation survey, sampling errors, and say, “continuous improvement” or other noninnovation business enhancers?
How can innovation and its impact on a business be measured? These are
weighting issues. NCSES and OECD are actively collecting basic questions that the cognitive testing project is considering.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 49

TABLE 4-2 Websites for Innovation Surveys from Regions Around the World
Region Countries Website
European Union (EU) 27 Austria, Belgium, Bulgaria, Cyprus, Czech Republic, http://ec.europa.eu/enterprise/policies/innovation/policy/
Denmark, Estonia, Finland, France, Germany, Greece, innovation-scoreboard/index_en.htm
Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg,
Malta, Netherlands, Poland, Portugal, Romania,
Slovenia, Slovakia, Spain, Sweden, UK

Rest of Europe Norway, Switzerland http://www.ssb.no/innov_en/


http://www.kof.ethz.ch/en/surveys/

Latin America Argentina, Brazil, Chile, Columbia, Cuba, Ecuador, http://www.ricyt.org


Mexico, Panama, Peru, Trinidad and Tobago, Uruguay

Asia Japan, Korea, Malaysia http://anstep.net/indicators.htm


http://www.kistep.re.kr/en/c3/sub3.jsp
http://www.kistep.re.kr/eng/publication/survey.jsp

Pacific Australia, New Zealand http://www.ausinnovation.org/articles/new-thinking-new-


directions-building-innovation-capability-in-australia.html
http://www.msi.govt.nz/get-connected/science-and-
innovation-in-nz/

Other Canada, South Africa http://www.statcan.gc.ca/concepts/index-eng.htm


http://www.hsrc.ac.za/en/departments/cestii/innovation-
survey
SOURCE: Adapted from Hall and Jaffe (2012).

TABLE 4-3 Percentage of Firms Reporting Innovation, BRDIS 2008-2011

Type of Innovation 2006-2008 (%) 2007-2009 (%) 2008-2010 (%) 2009-2011 (%)
Goods 4.6 5.8 5.5 5.4
Services 6.9 9.1 7.8 6.8
Manufacturing/production methods 3.6 4.4 4.1 3.8
Logistics/delivery/distribution methods 3.1 4.1 3.7 3.2
Support activities 6.9 9.3 8.2 7.2

NOTE: In the 2008, 2009, 2010, and 2011 BRDIS, companies were asked to identify innovations introduced in 2006-2008, 2007-2009, 2008-2010, and
2009-2011, respectively.
SOURCE: Tabulations from the 2008-2011 BRDIS.

of innovative outputs. This gap illustrates the importance of First, surveys are time-consuming and expensive for the
BRDIS, and it also suggests that these agencies, to the extent companies asked to complete them, as well as for those that
possible, should be working together (see Recommendation must compile, validate, and publish the results. The quality
8-2 in Chapter 8). of the survey depends not only on the number of responses,
but also on the amount of time and level of personal atten-
tion paid to them by respondents, which surely vary across
Limitations of Innovation Surveys
companies. Surveys of individuals, such as those relating to
The panel strongly supports the increased use of innova- the career paths of scientists and engineers discussed later in
tion surveys as a way to measure the output of the innovation this report, are becoming increasingly problematic because
system. Later in this chapter, new, nontraditional methodolo- of declining response rates.11 NCSES also experiences
gies for measuring innovation are discussed. In this section,
11See
however, the panel underscores the inherent limitations of the National Research Council (2013a) report on survey nonre-
any survey of innovation inputs and outputs. sponse. The report finds that “for many household surveys in the United
States, response rates have been steadily declining for at least the past two
decades. A similar decline in survey response can be observed in all wealthy

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

50 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

delays in obtaining data on firms, as is clear from the delays search activity will not show up as an increase in consumer
in publishing statistics from BRDIS. expenditures (Mandel, 2012).
A second limitation of innovation surveys is that the Sixth, although economists have long viewed innovation
nature of the economy is rapidly changing, largely as a result as unfailingly positive, extending the so-called “production-
of innovation itself. As a consequence, new and interesting possibility” frontier, recent events suggest that this is not
subjects (such as outsourcing of R&D or the rise of “big always the case. Various (but not all) financial innovations,
data”)12 on which survey questions and answers would be for example, had perverse effects that contributed to the
desirable are constantly coming to the attention of policy financial crisis of 2007-2008, which in turn led to the Great
makers and researchers. Yet because of the many lags built Recession (see, e.g., Litan, 2010). Likewise, while innova-
into the survey process—from survey design, to survey tions and communication that have facilitated the growth of
deployment, to compilation and validation of results, some- trade can make some countries better off, some countries or
times across multiple agencies—results may be stale or of portions of their population (unskilled workers in particular)
limited use once they are published. Timing also is impor- may be made worse off in a globalized economy (see, e.g.,
tant because rates of dissemination of new knowledge and Samuelson, 1962).
products are important variables of interest. Furthermore, in Finally, the virtue of innovation surveys is that they
the current budgetary environment, it will be difficult if not provide an overall picture of firms’ innovation activity.
impossible for NCSES to mount new surveys in the future However, experience suggests that the subjective nature of
without cutting back on those it already undertakes. survey responses poses some difficulties for interpretation.
Third, large, multinational companies that report R&D In particular, the percentage of self-reported innovative firms
expenditures report where they conduct the R&D (and thus varies across countries in unexpected ways. According to the
spend money on salaries, equipment, and the like). They 2009 BRDIS, for example, only 6.4 percent of U.S. firms
rarely if at all, however, report to the public and government had introduced new or significantly improved goods in the
statistical agencies the impacts of their R&D on their opera- previous 3 years, while only 10.3 percent reported new or
tions, and specifically the extent to which the location of significantly improved services. By contrast, the percentage
the R&D leads (or does not lead) to the creation of jobs and of German firms reporting themselves as innovative in the
income in the United States. period 2005-2007 was far higher, at 79.9 percent.13 In part,
Fourth, the traditional classification of economic transac- this large and surprising disparity may reflect differences in
tions into goods and services can miss important innovations survey methodology and questions. Even within the Euro-
that do not fall easily into either of these categories. A good pean Union (EU), however, unexpected differences are seen,
example is the rise of data as a separate economic category or with Germany reporting far higher rates of innovation than
product. Innovations in constructing and analyzing databases the Netherlands (44.9 percent) and the United Kingdom
are becoming increasingly important in an ever-expanding (45.6 percent).
range of industries—not only Internet search, but also retail- The panel does not have recommendations for NCSES
ing and manufacturing and now even health care. These inno- for responding to each of these challenges. Rather, the chal-
vations are important, but their economic value is difficult to lenges are described here to give a sense of how changes in
measure and categorize. the economy lead to difficulties in measuring innovation and
Fifth, a more general challenge is that some innovations— its impact. The panel does have some thoughts about general
the precise share is unknown—do not result in increased processes NCSES can follow to stay abreast of these changes
output, at least as conventionally measured. Accordingly, and meet these challenges, which are outlined later in this
“production functions” that relate R&D to increased output and subsequent chapters of the report.
may not be capturing well the real economic impact (or spill-
over effects) of some innovations. For example, the activity
IMPROVEMENTS TO BRDIS
of someone in Germany downloading a free application writ-
ten in the United States will not show up in either the U.S. This section examines improvements that could be made
trade or output statistics. Similarly, an increase in Internet to BRDIS in five areas: (1) international comparability; (2)
deeper information on innovations; (3) extensions to cover
organizational and marketing innovations, unmarketed
innovations, and a broader array of inputs to innovation;
countries” (National Research Council, 2013a, p. 1). The report also docu- (4) improvements to the presentation of information; and
ments declining response rates for a variety of surveys, including panel and
longitudinal surveys (pp. 14-30).
(5) better linkages between BRDIS data and other datasets.
12The term “big data” technically refers to a collection of data so large

and complex that it is difficult to store in one place and difficult to manage.
Recently, however—and in this report—the term has come to refer to data
that are gathered from sources other than surveys, such as administrative
records, websites, genomics, and geographic sensors, to name a few. Statisti- 13The U.S. statistic was calculated from the NSF summary sheet on
cally, the term and its characteristics are still being codified. BRDIS. The statistics for German firms are found in Eurostat (2010).

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 51

International Comparability tant innovation, and experts then classified these examples
according to whether they met the requirements for an inno-
One impediment to understanding and assessing innova-
vation. The study results provide valuable information on
tion in the United States is the lack of comparability between
how respondents perceive innovations and on what innova-
U.S. STI indicators and those developed by other countries.
tion indicators represent. They also indicate which types of
Comparability of BRDIS and CIS data requires sur-
innovations are deemed most important for business perfor-
veying similar populations of firms in comparable size
mance; in many cases, examples of new organizational meth-
categories and using similar techniques to derive estimates
ods and new marketing concepts and strategies were cited.
of innovation from the raw data collected. Also needed for
NCSES might want to consider including an open ques-
comparability are statistics using the same set of industries
tion in BRDIS on the most important innovation or innova-
typically used in statistics for other countries. An example
tions produced by the firm over the previous 3 years. Alter-
is the core set of industries used by Eurostat for comparison
natively, NCSES might want to consider commissioning a
of innovation statistics among EU countries, including min-
study similar to that of Arundel and colleagues (2010) on a
ing; manufacturing; and selected service industries, such
subset of responses to BRDIS to determine what firms are
as wholesale trade, transport services, financial services,
measuring as innovation. It would be useful to have firms
information technology (IT) services, R&D services, and
of different sizes, different sectors, and different geographic
business services. These data could be used to compile a
locations represented in such a study.
simple indicator of the share of product-process innovative
firms, defined as firms that have implemented a product or
process innovation. Extensions to BRDIS
There are three important ways in which BRDIS could be
Deeper Information on Innovations extended to make it more useful for policy purposes. First,
the innovation questions could be broadened to include orga-
Even with more comparable statistics on innovation, it
nizational and marketing innovations. Second, it might be
still will not be clear to users that firms are representing the
appropriate for policy makers if the survey included the cat-
same or similar things when they report product or process
egory of “unmarketed” innovations, reflecting the reality that
innovations. The BRDIS questions do not give enough infor-
an innovation may be ready to be marketed but be held back
mation to provide a full understanding of what the resulting
by such factors as regulation, financial constraints, or the
data and statistics mean. For example, users have no inde-
market power of incumbents. Third, the survey could track
pendent measure of whether the innovations of firms that
a broader array of inputs to innovation, including non-R&D
innovate but do not conduct R&D are more or less important
investments such as design, market research, and databases.
than those of firms that conduct R&D. Users would have
more confidence in and understanding of the BRDIS innova-
tion measures if they knew that knowledge input measures Organizational and Marketing Innovations
were correlated with actual performance. Users would have
The communications sector, broadly defined to include
even more confidence in the measures if they knew what
communications-related hardware and software as well as
firms meant by innovation—how closely company reporting
telecommunications providers, has clearly been one of the
on the survey matched NCSES’s definition of innovation.
most innovative sectors of the economy in recent years.
Without greater detail on specific innovations, moreover,
Within this sector, smartphones are a prominent example
the surveys paint an exceedingly broad picture of innova-
of product innovation, while increases in mobile broadband
tion. Knowing, for example, that 60 percent of a country’s
speed exemplify process innovation. Other recent innova-
firms have introduced some type of innovation does not help
tions in the communications sector, however, do not fit so
in understanding why and how innovation happened, what
neatly into the product and process categories. These innova-
impacts it has on the economy, and how it can be encouraged.
tions include social media, such as Twitter, Facebook, and
Indicators should provide not only a level but also insight into
LinkedIn; search, exemplified by Google and Microsoft;
how that level was achieved. Microdata from innovation sur-
web marketing and advertising, exemplified by Google’s
veys connected with other data on a firm might help achieve
AdSense/AdWords technology (see Box 4-4); and the avail-
this goal, but this approach has as yet not been exploited.
ability of mobile applications through Apple’s App Store.
Most innovative firms introduce both product and process
To identify such successful and important innovations as
innovations, as well as organizational or marketing innova-
AdSense/AdWords, NCSES may need to broaden the scope
tions (discussed below), and the impacts of the innovations
of the innovation questions on BRDIS to include organiza-
are likely to depend on many other business decisions (see
tional and marketing innovations as identified by the Oslo
OECD, 2010; Stone et al., 2008).
Manual (see Box 4-5). In addition, NCSES might consider
In a recent study conducted in Australia (Arundel et al.,
introducing a question about significant new data algorithms
2010), respondents were asked to describe their most impor-
as a category of innovation. Such a question would require

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

52 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

in the case of health-related innovations that require long


and expensive testing and are dependent on the willingness
BOX 4-4
Anatomy of a Marketing Innovation of Medicare and insurance companies to reimburse. For
example, pharmaceutical companies avoided bringing vac-
Google’s AdSense/AdWords advertising technology is one of cines to market because the reimbursement for them was
the most successful marketing innovations ever achieved. This relatively low, and the potential exposure to lawsuits was
technology, originally implemented piecemeal from 2000 to 2003, relatively high (Offit, 2005).
allows Google to place advertisements automatically on its own Third, in some industries, the market power of incumbents
webpages or those of partner sites, depending on the webpages’ may be an important deterrent to the commercialization of
content. At the same time, an advertiser can bid for the right to
be included in these placements. A retailer selling umbrellas, for
example, can bid for the right to place an advertisement on any
webpage using the term “rain.”
The AdSense/AdWords technology was clearly an innovation BOX 4-5
and proved remarkably difficult for competitors such as Yahoo! Four Forms of Innovation
and Microsoft to match. Moreover, the technology allows Google
to monetize successfully such product innovations as Gmail. The Oslo Manual (OECD-Eurostat, 2005, pp. 48-52) identifies
four forms of innovation. The 2012 Community Innovation Survey
SOURCE: Edelman et al. (2007). (Eurostat, 2012) employs the following definitions of these forms
of innovation:

Product: A product innovation is the market introduction of a new


or significantly improved good or service with respect to its capa-
significant new conceptual development, but would help
bilities, user friendliness, components or sub-systems. Product
capture technological developments in some of the most
innovations (new or improved) must be new to your enterprise,
dynamic sectors of the economy.
but they do not need to be new to your market. Product innova-
tions could have been originally developed by your enterprise or
Unmarketed Innovations by other enterprises or institutions. A good is usually a tangible
object such as a smartphone, furniture, or packaged software, but
A second potential expansion of the BRDIS innovation
downloadable software, music and film are also goods. A service
questions could be extremely useful to policy makers. As
is usually intangible, such as retailing, insurance, educational
noted earlier, according to the current definition of innova-
courses, air travel, consulting, etc.
tion, a product must be bought to market, or implemented,
to be considered an innovation. A new product, for example,
Process: Process innovations must be new to your enterprise,
does not count as an innovation unless it can be purchased
but they do not need to be new to your market. The innovation
or used by a consumer or a business.
could have been originally developed by your enterprise or by
From the perspective of the macroeconomic impact of
other enterprises, and excludes purely organizational innovations
innovation on growth and productivity, such a requirement
makes sense. An innovation cannot affect economic perfor-
Organizational: An organizational innovation is a new organi-
mance unless it can be marketed or implemented. It is also
zational method in your enterprise’s business practices (including
important to understand, however, why some industries and
knowledge management), workplace organization or external
some countries appear to produce innovations at a rapid
relations that has not been previously used by your enterprise. It
pace, while others are less productive in turning innovative
must be the result of strategic decisions taken by management,
activities into marketable innovations. Part of the explana-
and excludes mergers or acquisitions, even if for the first time.
tion is that promising innovations can be stalled by a variety
of factors.
Marketing: A marketing innovation is the implementation of a
First, a tightening of approval regulations for scientific,
new marketing concept or strategy that differs significantly from
political, or legal reasons can result in fewer innovations
your enterprise’s existing marketing methods and which has not
coming to market, even if the pace of technological advance
been used before. It requires significant changes in product design
stays the same.14
or packaging, product placement, product promotion or pricing,
Second, the nature of payment systems can have a simi-
and excludes seasonal, regular and other routine changes in
lar dampening effect on marketable innovation, especially
marketing methods.

14The panel notes that some regulations achieve societal goals that SOURCE: The 2012 Community Innovation Survey (Eurostat, 2012).
increase social well-being and that advancing innovation clearly is not the
only means of advancing a society’s goals.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 53

potential innovations. For example, if companies lack access improved products or processes (excluding expendi-
to important sales channels that are controlled by incum- tures on equipment for R&D);
bents, then they may not find it worthwhile to commercialize • acquisition of existing know-how, copyrighted
a new product even if that product would be economically works, patented and non-patented inventions, etc.
viable in a more competitive market. from other enterprises or organisations for the devel-
Finally, in some cases, a potential innovation may not be opment of new or significantly improved products
introduced to market because of the need for other, comple- and processes;
mentary innovations. A classic example is Chemcor, an • in-house or contracted out training for your personnel
ultrahard glass invented in the 1960s by Corning. Because specifically for the development and/or introduc-
of the cost of production, Chemcor did not find a place in tion of new or significantly improved products and
the market until 2006. Renamed Gorilla Glass, it became the processes;
product of choice for cell phone screens. • in-house or contracted out activities for the market
The concept of unmarketed innovations is implicit in an introduction of your new or significantly improved
existing question on the CIS: goods or services, including market research and
launch advertising;
During the three years 2010 to 2012, did your enterprise have • in-house or contracted out activities to design or alter
any innovation activities that did not result in a product or the shape or appearance of goods or services; and
process innovation because the activities were • other in-house or contracted out activities to imple-
ment new or significantly improved products and
• abandoned or suspended before completion processes such as feasibility studies, testing, tooling
• still ongoing at the end of 2012
up, industrial engineering, etc.” (Eurostat, 2012,
p. 6).15
The panel believes NCSES should consider adding a
similar question to BRDIS. NCSES might also consider While having reliable data on these innovation-related
asking respondents to rank the main reasons why the outputs expenditures would be useful, it may be difficult for com-
of their innovation activities have not yet been marketed panies to report these expenditures if they do not already
or implemented, including not yet ready, lack of funding, record them in their accounts. Interviews and testing with
lack of sufficient skilled labor, the need to meet regulatory companies would help in discerning which types of expen-
requirements, or blocked from market by competitors. The ditures can be reported reliably. Expenditures on training
panel acknowledges that such questions may be difficult for and design may be among the most feasible to measure and
respondents to answer, although such a question on factors are important for non-R&D innovation activities. Data on
hampering product and process innovation has been asked innovation-related expenditures may also provide useful
in the CIS for some years (Eurostat, 2012). If such questions input for the development of statistics on knowledge-based
were added to BRDIS, then the insights they would yield into capital and “intangible assets” (discussed in Chapter 5).
bottlenecks for innovation could be quite useful.

Improvements to the Presentation of Information


A Broader Array of Inputs to Innovation
NCSES has long been focused on providing information
The panel is aware of the budget constraints faced by on levels of R&D funding by performing and funding sec-
NCSES. With these constraints in mind, the panel notes that tor. The agency’s core tables on R&D report dollar amounts
BRDIS could collect other types of data that would be helpful of R&D, both nominal and real, and R&D as a share of
for policy makers. Hall and Jaffe (2012), whose paper was gross domestic product (GDP). In addition, BRDIS collects
commissioned for this study, note that it would be helpful a wealth of other policy-relevant information that has the
to have more information on the amount spent on different potential to provide guidance for policy makers. In February
kinds of innovation activities. For example, BRDIS currently 2012, for example, NCSES published an InfoBrief on busi-
asks respondents whether they “acquired more than 50 per- ness use of intellectual property.
cent ownership in another company for the primary purpose At the same time, NCSES could greatly improve the
of acquiring their IP [intellectual property].” However, it usefulness of its surveys by quickly publishing more “cross-
would also be useful to know how much the company spent tab” tables on key policy issues, or by making it easier for
on the acquisition of external knowledge, marketing and researchers and policy makers to access quickly the underly-
design of the improved goods or services, and other innova-
tion activities. These questions are drawn from question 5.1
of the CIS, which asks specifically about: 15There are some differences between the 2010 and 2012 CIS for this
and other questions. The 2012 version includes copyrighted works under
• “acquisition of advanced machinery, equipment, soft- acquisition of existing knowledge. It also clarifies that the activities may
ware and buildings to be used for new or significantly be in-house or contracted.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

54 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

ing data necessary to construct such tables.16 A crosstab table


shows the interrelationship of two variables. For example,
BOX 4-6
BRDIS collects information on the amount of research Examples of Questions That Could Be
spending outside the United States by U.S.-based compa- Informed by Policy-Relevant Crosstab Tables
nies. That information is important in itself, and NCSES has
published the breakdown of foreign versus domestic R&D Compared with other companies:
spending for different industries. From the perspective of
policy, however, it is also useful to know whether companies • Are companies that perform research overseas more likely to
that conduct a higher share of their R&D overseas relative report innovations?
to other companies are more or less likely to report innova- • Are companies that perform research in California more likely
tions. That is, is globalization of R&D linked with a higher to report innovations?
or lower propensity to innovate? Such a table, if published by • Are companies that collaborate with other companies more
NCSES, could help guide policy makers, stimulate research, likely to report innovations?
and inform public debate on U.S. innovation policies. • Are companies that partner with academic institutions more
likely to report innovations?
RECOMMENDATION 4-2: The National Center for • Are companies that have a high percentage of science, tech-
Science and Engineering Statistics should build on its nology, engineering, and mathematics personnel more likely
Business Research and Development and Innovation to report innovations?
Survey (BRDIS) to improve its suite of innovation • Are companies that acquire intellectual property more likely to
indicators in the following ways: report innovations?
• Are companies in regulated industries more likely to report
• t abulate the results from BRDIS using the same unmarketed innovations?
cutoffs for firm size (as well as comparable indus- • Are companies engaged in health care research more likely to
try sectors) that are used by OECD countries in report unmarketed innovations?
order to facilitate international comparisons; • Are companies created to commercialize academic research
• fund research exploring precisely what compa- more likely to report unmarketed innovations?
nies mean when they report an innovation or
report no innovation on BRDIS—such research
would help inform current policy debates;
• broaden the innovations tracked by BRDIS to
encompass organizational and marketing inno- access to the raw data, it is relatively easy for NCSES to
vations, as well as new data algorithms; construct high-interest crosstab tables and publish them
• consider adding a section to BRDIS on unmar- quickly. Such a process is important to the agency’s mission
keted innovations, giving respondents the oppor- of providing statistics that are relevant to policy makers
tunity to cite the main reason these innovations and the public.
have not yet been marketed or implemented; The panel also found that NCSES’s limited access to some
• as funds permit, extend BRDIS to gather infor- datasets constrains the timely development of innovation
mation on innovation-related expenditures in indicators. For instance, some data are available at the U.S.
such areas as training and design; and Census Bureau’s research data centers before NCSES has
• publish more results from BRDIS that link inno- on-site access to those data at its Survey Sponsor Data Cen-
vation to business characteristics, including the ter (SSDC). This is the case for the BRDIS data, which the
amount of research and development spending Census Bureau collects on behalf of NCSES. Ready access to
by U.S.-based companies outside of the United these data is imperative if NCSES is to satisfy the demands
States. Production and distribution of such of users for timely innovation statistics. Housing the BRDIS
cross-tabulations should be timely, and they data in the SSDC would be one key way to improve the time-
should address contemporary policy questions. liness of the statistical analysis of the data and the publication
of related R&D and innovation indicators.
The globalization of research is only one example of
an area in which presentation of a crosstab table could be RECOMMENDATION 4-3: The Survey Sponsor
helpful for policy makers. Box 4-6 identifies examples of Data Center at the National Science Foundation
policy-relevant questions for which a crosstab table could should house the Business Research and Develop-
be useful, especially if published in a timely manner. With ment and Innovation Survey data, improving access
to the data for National Center for Science and Engi-
16Timeliness and relevance are both data quality measures (see Recom- neering Statistics staff who develop the research and
mendation 8-1 in Chapter 8). development statistics.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 55

Linkages Between BRDIS Data and Other Datasets of the United States on an aspect of the country’s innovation
capacity.
The data gathered in BRDIS could be used to begin
NCSES has a unique set of data in BRDIS, which, if com-
developing statistics on high-growth firms and “gazelles.”
bined with other datasets, could be instrumental in answering
The Manual on Business Demography Statistics (OECD-
these and other important questions. Integrating data on firm
Eurostat, 2008, Chapter 8, p. 61) defines high-growth enter-
dynamics (and the related employment effects) would take
prises as “all enterprises with average annualised growth
time and resources. During his presentation at the workshop,
greater than 20% per annum, over a three year period. . . .
Haltiwanger described three Census Bureau datasets that,
A size threshold has been suggested as 10 employees at the
together with BRDIS data, would allow NCSES to develop
beginning of the growth period.” Gazelles are the subset of
indicators of business dynamics:
high-growth enterprises that are up to 5 years old (OECD-
Eurostat, 2008, Chapter 8, p. 63). These thresholds are arbi-
• Longitudinal Business Database—tracks all estab-
trary and based only on convention. NCSES could conduct
lishments and firms with at least one employee,
its own sensitivity analysis to fine-tune the definitions of
including start-ups, from 1976 to the present;
high-growth firms and gazelles.17
• Integrated Longitudinal Business Database—tracks
During the panel’s July 2011 workshop, several speak-
all nonemployer firms and integrated-with-employer
ers18 mentioned the importance of tracking trends in the
firms from 1994 to the present; and
sustainability of jobs in these types of firms during economic
• Longitudinal Employer-Household Dynamics—
downturns (even if total employment is small). It would also
tracks longitudinally all employer-employee matches
be useful to have firm data by age classes to determine over
and transitions (hires, separations, job creation, and
time whether high-growth firms or gazelles in particular
job destruction) from 1990 to the present.
have a higher incidence of innovation activity relative to
other firms. In his presentation at the July 2011 workshop,
Questions from the Census Bureau’s 2007 and 2012 Eco-
Hollanders showed that high-growth firms are significantly
nomic Census, Company Organization Survey, and Man-
more innovative than other firms in his dataset. The connec-
agement and Organizational Practices Survey can also yield
tion between high-growth firms and innovation is complex,
useful information on R&D and other innovation activities
and these data would help researchers better understand it.
for establishments. In addition, infrastructure datasets can
Statistics on high-growth firms and gazelles could also be
track relationships between start-up and young high-growth
used to answer the question of whether these types of firms
firms and large, mature firms, and can be linked further to
drive economic and job growth. A simple table could com-
patent and citation data. Important as well is to link data on
pare the economic characteristics of high-growth and other
firm dynamics to those on innovation outputs, such as patent
firms that are and are not innovative, ideally over time.
and citation data.
At the panel’s September 2011 meeting in Washington,
Haltiwanger proposed that indicators track firm dynam-
DC, representatives from BLS, the U.S. Census Bureau,
ics by geography, industry, business size, and business age.
and BEA mentioned that linking certain datasets among
Hollanders noted that European countries and other OECD
them would yield reasonable numbers on gazelles. A table
members are continuing to fine-tune their measures of firm
with these numbers could be added to the Science and
dynamics. NCSES’s indicators on this dimension could
Engineering Indicators or become the foundation of an
further the international comparability of its STI indicators.
InfoBrief. The following indicators could be produced using
Building the foundations for indicators of firm dynamics
BRDIS and other data on high-growth firms and gazelles:
using BRDIS and other datasets would give NCSES a pro-
number of high-growth enterprises as a percentage of the
ductive platform for developing several STI indicators that
total population of active enterprises with at least n-number
are policy relevant.
of employees, and number of gazelles as a percentage of
Clearly, developing publishable statistics on high-growth
all active enterprises with at least n-number of employees
firms and gazelles is a multistage task requiring data acquisi-
that were born 4 or 5 years ago. These indicators would be
tion and linkage in addition to use of the data available from
comparable to those produced in several other countries, thus
BRDIS. A good first step would be for NCSES to explore
increasing users’ understanding of the comparative position
linking its BRDIS data with data on firm dynamics from
BLS.
17Petersen and Ahmad (2007) present a technique for conducting this
type of analysis in OECD (2007).
RECOMMENDATION 4-4: The National Center for
18Howard Alper, University of Ottawa; Robert Atkinson, Information Science and Engineering Statistics (NCSES) should
Technology and Innovation Foundation; John Haltiwanger, University begin a project to match its Business Research and
of Maryland; Hugo Hollanders, United Nations University’s Maastricht Development and Innovation Survey data to data
Economic and Social Research Institute on Innovation and Technology from ongoing surveys at the U.S. Census Bureau
(UNU-MERIT); and Brian MacAulay, National Endowment for Science,
Technology, and the Arts.
and the Bureau of Labor Statistics. It should use

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

56 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

the resulting data linkages to develop measures of ing activities,19 and (5) trademark databases.20 This section
activities by high-growth firms, births and deaths of examines the advantages and disadvantages of business prac-
businesses linked to innovation outputs, and other tice data from the perspective of NCSES. Further discussion
indicators of firm dynamics, all of which should be of the use of these types of data for developing STI indicators
tabulated by geographic and industry sector and by appears in Chapter 7 of this report.
business size and business age to facilitate compara-
tive analyses. NCSES should conduct a sensitivity
Conceptual Background
analysis to fine-tune meaningful age categories for
high-growth firms. The use of business practice data as a supplement to
traditional surveys is under active consideration in statisti-
cal agencies. Robert Groves, then director of the Census
USE OF NONTRADITIONAL METHODOLOGIES
Bureau, recently delineated the key issues entailed in using
Traditionally, NCSES and its predecessors have used sur- these data (see Box 4-7), noting: “Combined, the ‘big data’
veys to trace the inputs and outputs of the innovation system. and the benchmark survey data can produce better statistics.”
As noted earlier, however, executing a survey is an expensive The advantages of business practice data include
and time-consuming process, requiring writing and testing
questions, identifying the universe of potential respondents, • Timeliness—Collecting and analyzing survey data
collecting the data, cleaning and validating the data, analyz- is a lengthy process. Because most business practice
ing the data, and then finally publishing the results. Thus, for data today are digital, they can be available for analy-
example, NCSES did not publish the topline R&D spending sis in near real time.
results from the 2009 BRDIS until March 2012. • Detail—Survey data offer limited detail by industry
Another issue with government surveys is that those and geographic location. Business practice data can
being surveyed may not respond, and when they do, they are be used to track innovation activity by detailed indus-
almost always guaranteed confidentiality. As a result, some try or subnational area.
or even many results must be withheld to avoid disclosing, • Flexibility—Survey methods require making some
even indirectly, the responses of individual companies. In the assumptions about the nature of innovation many
basic table showing R&D spending by industry and source years in advance. The ability to adjust the measuring
of funding from the 2010 and 2011 BRDIS (Table 2 in Info- tool easily is helpful, particularly when new catego-
Brief 13-335), for example, 37 and 36 (respectively) of the ries of innovation emerge.
154 cells are suppressed. The significance of this problem
grows when more detailed industries or subnational areas At the same time, business practice data have disadvantages
are considered. Take, for example, the interesting ques- that make them an imperfect substitute for conventional
tion of whether the location of R&D conducted by the IT surveys. These disadvantages include
industry is becoming more or less concentrated in Silicon
Valley. If BRDIS were to ask companies to apportion their • “Institutional drift”—Business practice data are
research spending by metro area, the data for most metro generated by normal business activity, but because
areas across the country would likely have to be suppressed patterns of business activity change over time, inter-
because information on a specific firm might be identified in pretations of business practice data are not necessar-
cases involving a small number of firms in a given sector or ily stable.
region. It might even be difficult to obtain useful data at the • Difficulty with cross-industry/cross-country
state level for the same reason. comparisons—Different industries and countries
Until fairly recently, there was no good alternative to may have very different business practices.
surveys for collecting data on innovation inputs and outputs.
Increasingly, however, businesses and individuals are gener-
ating detailed electronic data in the normal course of their
economic activity. Such data are available either in a firm’s 19“Altmetrics are alternative measures that can supplement citation counts

administrative records, publicly on the web, or from third and journal impact factors as measures of the impact of scholarly commu-
parties that collect them in the normal course of economic nications. Such measures are generally derived from online activity such as
activity, and can be obtained in digital form from a given mentions, downloads, tweets, blog posts, Facebook “likes,” bookmarking
and other similar evidence of attention” (Travis, 2013).
firm. Such nontraditional data are referred to here as “busi- 20Note that not all business practice data fall into the category commonly
ness practice” data. Examples of these data include several referred to as “big data.” For example, administrative records are included
datasets derived from the Internet: (1) reports on innovations, in the panel’s definition of business practice data but are not considered
(2) help-wanted ads, (3) journal articles on new products and “big data.” Also note that the term “big data” had not been formally codi-
services, (4) altmetric measures of scientific and engineer- fied by statistical agencies at the time this report was being written. Lastly,
“big data” is not a panacea (see Boyd and Crawford [2011] for important
caveats).

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

MEASURING INNOVATION 57

BOX 4-7 BOX 4-8


“And Now, for Something a Little Different” Examples of Web-Based Data on Innovation
Excerpt from a Blog Post by
Robert Groves, Director, U.S. Census Bureau Lists of Top Innovations
• http://www.fastcompany.com/1738506/the-10-most-
If we had access to customer purchase transactions volume, innovative-companies-in-health-care
we might construct models blending our benchmark sample • http://www.rdmag.com/Awards/RD-100-Awards/2011/06/
survey data with the continuous transaction data, to produce R-D-100-2011-Winners-Overview/
more timely and more disaggregated estimates. The strength of • Small Business Administration—http://www.sba.gov/content/
the transaction data will be their timeliness and the large number sba-announces-winners-2011-tibbetts-awards
of transactions they reflect; their weakness will be that they do not • Technology Review—http://www.technologyreview.com/tr50/
include many transactions conducted in ways other than those • h ttp://my.clevelandclinic.org/media_relations/library/
the data reflect (e.g., cash might be omitted). The strength of our 2011/2011-10-6-cleveland-clinic-unveils-top-10-medical-
benchmark survey will be its statistical coverage of the entire innovations-for-2012.aspx
population of business units; its weakness is its lack of timeliness
and its relatively small sample size of firms. Combined, the “big Innovation Data Reported by Companies
data” and the benchmark survey data can produce better statistics. • New England BioLabs products—http://www.neb.com/
Sometimes the link between our sample surveys and the big nebecomm/newprod.asp
data will be time, other times it will be space. “Big data” will be • GE products—http://www.ge.com/products_services/
useful for constructing small area estimates. For example, internet directory/by_product.html
sites listing asking prices for houses may be accompanied with • Corning—http://www.corning.com/displaytechnologies/en/
exact location of the units. Their strength is that they offer millions index.aspx
of records of prospective sales; their weakness is that they don’t
cover all areas of the country, not all sales are covered, and asking Site That Collects Innovation Data
prices are not sale prices. Our sample survey on residential sales
offers statistical coverage of all sales, but its sample size is too • ProductDb: http://productdb.org
small to provide statistics on all areas. Combining the two data
series might offer more spatial detail.
At other times, the link between the big data and our sample
survey data may be measures that are highly correlated to our key
statistics. For example, we might have access to traffic volume data
continuously streaming based on traffic cameras, with location jobs at a more detailed geographic and occupational
codes to permit fine spatial detail. Our sample survey reports of level than is possible with government survey data.
commuting times from home to place of work might be enhanced
by statistically combining them with the traffic count data from Examples of Business Practice Data21
available cameras. The strength of the traffic camera counts would
be very fine grain detail on time; the weakness would be coverage Understanding of innovation would be facilitated if
of all roads and counts of commercial traffic as well as private cars. NCSES regularly (annually or biannually) published a list
of significant innovations brought to market. Such a list
SOURCE: Available: http://directorsblog.blogs.census.gov/2012/06/27/and-now- would be useful for assessing trends in innovation and how
for-something-a-little-different/ [January 2014]. they relate to growth and jobs. Also helpful would be to have
a list of major potential innovations under development,
based on publicly reported data. Such lists have occasion-
ally been manually constructed in the past (see Box 4-8).
In 1982, for example, the Small Business Administration
As a result, using business practice data requires innovative sponsored a project that entailed gathering information
statistical techniques to standardize measures across time, from 46 trade magazines and identifying and coding 8,074
industries, and location. In addition, key indicators must be innovations—4,476 in manufacturing industries. This infor-
maintained consistently over time to provide a benchmark. mation included

RECOMMENDATION 4-5: The National Center • model name, trade name, or trademark;
for Science and Engineering Statistics should make • name and description of the innovation;
greater use of business practice data to track research
21Nonsurvey methods for extracting data for the development of STI
and development spending and innovation-related
indicators are discussed in greater detail in Chapter 7.

Copyright National Academy of Sciences. All rights reserved.


Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy

58 CAPTURING CHANGE IN SCIENCE, TECHNOLOGY, AND INNOVATION

• year of introduction; cators program with respect to measuring innovation. These


• innovation type (product or process); recommendations address four topics: (1) new and improved
• innovation significance (new class, new type, signifi- measures of innovation inputs and outcomes based on exist-
cant improvement, updated model); and ing data, with clear implications for economic performance;
• origin of technology and source of funding. (2) analysis, based on existing data, that is more comparable
across countries than is currently the case; (3) improved data
Today, more efficient techniques for creating lists of inno- resources and accessibility; and (4) augmented survey ques-
vation are available, based on web scraping. One can also tions that would ensure better reporting on innovation activi-
look at the propagation of technologies by examining the ties by firms. The panel believes NCSES should focus first
publication of product manuals. The panel considers these on activities that use existing data while further developing
techniques in greater detail in Chapter 7. capabilities for linking its data with those of other agencies
and using frontier tools to produce STI indicators in a more
timely fashion.
SUMMARY
In this chapter, the panel has offered five recommenda-
tions whose implementation would improve NCSES’s indi-

Copyright National Academy of Sciences. All rights reserved.

You might also like