The Use of Content Analysis in Quantitative Research

The Use of Content Analysis in Quantitative Research
___________________________
Dr. Timothy G. Pollock
Smeal College of Business The Pennsylvania State University
Academy of Management Annual Meeting

August 8, 2009
Presentation Outline
Role of Textual Data in Quantitative Analysis Sources of Data Selecting the Unit of Analysis Developing Concept Categories
Coding the Data
Role of Text in Quantitative Analysis

Narrative statements provide insights into the
cognitive processes of individuals
Published narratives are unobtrusive sources of

information that can be collected post hoc, and are generally less subject to retrospective sensemaking
Role of Text in Quantitative Analysis

Published narratives may be the primary way in which
management communicates with certain stakeholder groups
Concepts discussed in the narratives can be counted

and used as either independent or dependent variables
Sources of Textual Data

Interviews Open Ended Surveys Internal Documents (reports, memos, etc.) Required SEC Filings (proxies, annual reports) News Reports Speeches Blogs
Worth and Words Study

Examined factors influencing the frequency with which compensation committees used different justifications in explaining their executive pay practices Sample: 266 companies included in the S&P 500 in 1992 Narrative Source: Compensation Committee Reports in 1993 Proxy statements 12,500 sentences were content analyzed Four types of Justifications: External Validation Shareholder Alignment Accounting Perf. Measures Market Perf. Measures
Selecting a Unit of Analysis

Word
Sentence
Paragraph
Story
Developing Concept Categories

Three Strategies for Developing Concept Categories
Apply categories that have been previously defined

using a particular theoretical framework or conceptual schema
Begin with no predefined structure, and allow the

categories to emerge from the data
Combination Approach - start with a partially defined

set of categories, and modify, add, or delete categories as the analysis proceeds
Coding the Data - CATA

Concepts are identified based on the use of particular
words and phrases
The Hardest Part: Identifying the words and phrases

that capture the concepts you want and avoid the concepts you dont
Sources of words and phrases brainstorming general uses thesaurus vocabulary lists generated by your software fuzzy searches

ZyIndex
Uses Boolean logic to identify phrases Identifies and presents all hits in context provides a vocabulary list with frequency counts
Example: Identify all variations of return to stockholders
return* w/in 3 stock*
returns to stockholders return to stockholders return to stockholder return on stockholders return on stockholders return on stockholder return to stockowner return on stockowner return on stockowners return on stockowners return to stockowners

VBPro
Analyzes text and provides concept counts based upon

dictionaries of terms which you develop for each concept
Free!!! (http://www.yoshikoder.org/vbpro.html)
Provides both excerpted text with the concept marked and

quantitative data which is easily imported into statistical analysis packages
Quantitative output can be either occurrence (1,0) or count Counts provided at either the sentence, paragraph or story
level of analysis
Frequency Counts of Dependent Variable Concept Words
CONCEPT EXTERNAL VALIDATION advisors consultant consultants consultant's consulting hewitt hewitt's Perrin tpf&c ors hay experts survey surveys surveyed
FREQUENCY 631 5 92 137 5 73 40 3 12 2 2 6 5 118 111 20
CONCEPT SHAREHOLDER shareholder shareholders shareowner shareowners stockholder stockholders ALIGN link linking linked tie tying tied align aligned aligning interest interests
FREQUENCY 1399 418 455 15 19 180 312 1505 129 39 82 42 12 109 523 26 38 138 367
CONCEPT ACCOUNTING PERFORMANCE roa roe eps ros rotc return on assets return on equity return on investment return on capital return on sales return on total capital return on average equity net income earnings cash flow credit rating ratio ratios return on sales/revenues return on targeted equity return on expense return on common equity return on shareholders ' equity return on average common equity return on opening equity return on average equity returns on the company's assets return on beginning - of - year equity return on average shareholder equity operating profits operating profit operating income operating margins
FREQUENCY 1531 19 97 82 7 9 42 164 21 29 17 7 3 172 558 127 1 36 9 1 1 1 10 11 3 1 3 1 1 3 7 19 65 4

Example Output for Shareholder Alignment
#AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , #AZA , 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0 0 0 0 0 0 0 1 1 0 0 0 0 3 0 1 0 0 0 0 1 0 0 0 0 0 0 0 2 0 0
Alza uses a shareholder alignment justification 4 times

The concepts coded must also be validated Difficult to get 100% inclusion of intended concepts
and 100% exclusion of false hits
Validation done manually against a sample of the

machine coded text
80% or better accuracy on concept hits No more than 5% of coded sentences false hits Misses assumed to be randomly distributed

External Validation
The corporation, in establishing base salaries, levels of incidental and/or supplemental compensation, and incentive compensation programs for its officers and key executives, assesses periodic compensation SURVEY surveys and published data covering the electrical/electronics industry and industry in general.
Shareholder Alignment
In this way, long-term compensation is ALIGN tied to increased OWNERS shareholder value and long-term service to the corporation.
Accounting Measures of Performance

In reviewing Mr. Bossidy's performance, the committee focused primarily on the company's performance in 1992, which exceeded goals with very substantial improvement in operating ACCTMEAS earnings per share and ACCTMEAS cash flow.
Our Findings Worth & Words

The frequency with which these justifications are used
is affected by the level of CEO pay, company performance, and the presence of concentrated and active shareholders
Elements of pay, especially bonus compensation, and

company performance interact with shareholder characteristics to influence the frequency with which different justifications are used
Companies used justifications both to explain away

high levels of pay, and to highlight when executive compensation is not paid
Coding the Data Celebrity

42,000+ articles from Business Week on 291 companies (1991-2004)
Used Linguistic Word Count software provides validated dictionaries of positive emotion and negative emotion words
Coded article as positive if >60% affective language positive; negative if >60% affective language negative; otherwise neutral Calculated Janis-Fadner Coefficient of Imbalance for each year Coded celebrity = 1 if in top 25% in both number of articles and JF Coefficient in the same year

The Use of Content Analysis in Quantitative Research

Uploaded by

Copyright:

Available Formats

The Use of Content Analysis in Quantitative Research

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The Use of Content Analysis in Quantitative Research

Uploaded by

Copyright:

Available Formats

The Use of Content Analysis in Quantitative Research

Academy of Management Annual Meeting

Coding the Data

Role of Text in Quantitative Analysis

Published narratives are unobtrusive sources of

Role of Text in Quantitative Analysis

Concepts discussed in the narratives can be counted

Sources of Textual Data

Worth and Words Study

Selecting a Unit of Analysis

Developing Concept Categories

Apply categories that have been previously defined

Begin with no predefined structure, and allow the

Combination Approach - start with a partially defined

Coding the Data - CATA

The Hardest Part: Identifying the words and phrases

Coding the Data - CATA

Coding the Data - CATA

Analyzes text and provides concept counts based upon

Provides both excerpted text with the concept marked and

Frequency Counts of Dependent Variable Concept Words

FREQUENCY 631 5 92 137 5 73 40 3 12 2 2 6 5 118 111 20

FREQUENCY 1531 19 97 82 7 9 42 164 21 29 17 7 3 172 558 127 1 36 9 1 1 1 10 11 3 1 3 1 1 3 7 19 65 4

Coding the Data - CATA

Alza uses a shareholder alignment justification 4 times

Coding the Data - CATA

Validation done manually against a sample of the

Coding the Data - CATA

Accounting Measures of Performance

Our Findings Worth & Words

Elements of pay, especially bonus compensation, and

Companies used justifications both to explain away

Coding the Data Celebrity

You might also like