SUGI 27 Data Mining Techniques

Paper 114-27

Predicting Customer Churn in the Telecommunications

â
Industry –– An Application of Survival Analysis Modeling Using SASâ

Junxiang Lu, Ph.D.

Sprint Communications Company
Overland Park, Kansas

churn, they are able to design customized customer

ABSTRACT
communication and treatment programs in a timely
Conventional statistical methods (e.g. logistics regression, efficient manner.
decision tree, and etc.) are very successful in predicting
customer churn. However, these methods could hardly Conventional statistical methods (e.g. logistics regression,
predict when customers will churn, or how long the decision tree, and etc.) are very successful in predicting
customers will stay with. The goal of this study is to customer churn. These methods could hardly predict
apply survival analysis techniques to predict customer when customers will churn, or how long the customers
churn by using data from a telecommunications company. will stay with. However, survival analysis was, at the very
This study will help telecommunications companies beginning, designed to handle survival data, and therefore
understand customer churn risk and customer churn is an efficient and powerful tool to predict customer
hazard in a timing manner by predicting which customer churn.
will churn and when they will churn. The findings from
this study are helpful for telecommunications companies
OBJECTIVES
to optimize their customer retention and/or treatment
The objectives of this study are in two folds. The first
resources in their churn reduction efforts.
objective is to estimate customer survival function and
customer hazard function to gain knowledge of customer
INTRODUCTION churn over the time of customer tenure. The second
In the telecommunication industry, customers are able to objective is to demonstrate how survival analysis
choose among multiple service providers and actively techniques are used to identify the customers who are at
exercise their rights of switching from one service high risk of churn and when they will churn.
provider to another. In this fiercely competitive market,
customers demand tailored products and better services at
less prices, while service providers constantly focus on DEFINITIONS AND EXCLUSIONS
acquisitions as their business goals. Given the fact that This section clarifies some of the important concepts and
the telecommunications industry experiences an average exclusions used in this study.
of 30-35 percent annual churn rate and it costs 5-10 times
more to recruit a new customer than to retain an existing Churn – In the telecommunications industry, the broad
one, customer retention has now become even more definition of churn is the action that a customer’s
important than customer acquisition. For many incumbent telecommunications service is canceled. This includes
operators, retaining high profitable customers is the both service-provider initiated churn and customer
number one business pain. Many telecommunications initiated churn. An example of service-provider initiated
companies deploy retention strategies in synchronizing churn is a customer’s account being closed because of
programs and processes to keep customers longer by payment default. Customer initiated churn is more
providing them with tailored products and services. With complicated and the reasons behind vary. In this study,
retention strategies in place, many companies start to only customer initiated churn is considered and it is
include churn reduction as one of their business goals. defined by a series of cancel reason codes. Examples of
reason codes are: unacceptable call quality, more
In order to support telecommunications companies favorable competitor’s pricing plan, misinformation given
manage churn reduction, not only do we need to predict by sales, customer expectation not met, billing problem,
which customers are at high risk of churn, but also we moving, change in business, and so on.
need to know how soon these high-risk customers will
churn. Therefore the telecommunications companies can High-Value Customers – Only customers who have
optimize their marketing intervention resources to prevent received at least three monthly bills are considered in the
as many customers as possible from churning. In other study. High-value customers are these with monthly
words, if the telecommunications companies know which average revenue of $X or more for the last three months.
customers are at high risk of churn and when they will If a customer’s first invoice covers less than 30 days of
SUGI 27 Data Mining Techniques

service, then the customer monthly revenue is prorated to PROC PHREG has gained popularity over PROC
a full month’s revenue. LIFEREG in the last decade since it handles time-
dependent covariates. However if the shapes of survival
Granularity – This study examines customer churn at the distribution and hazard function are known, PROC
account level. LIFEREG produces more efficient estimates (with smaller
standard error) than PROC PHREG does.
Exclusions – This study does not distinguish international
customers from domestic customers. However it is SAMPLING STRATEGY
desirable to investigate international customer churn On August 16, 2000, a sample of 41,374 active high-value
separately from domestic customer churn in the future. customers was randomly selected from the entire customer
Also, this study does not include employee accounts, since base from a telecommunications company. All these
churn for employee accounts is not of a problem or an customer were followed for the next 15 months.
interest for the company. Therefore August 16, 2000 is the origin of time and
November 15, 2001 is the observation termination time.
SURVIVAL ANALYSIS AND CUSTOMER CHURN During this 15-month observation period, the timing of
Survival analysis is a clan of statistical methods for customer churn was recorded. For each customer in the
studying the occurrence and timing of events. From the sample, a variable of DUR is used to indicate the time that
beginning, survival analysis was designed for longitudinal customer churn occurred, or for censored cases, the last
data on the occurrence of events. Keeping track of time at which customers were observed, both measured
customer churn is a good example of survival data. from the origin of time (August 16, 2000). A second
Survival data have two common features that are difficult variable of STATUS is used to distinguish the censored
to handle with conventional statistical methods: censoring cases from observed cases. It is common to have
and time-dependent covariates. STATUS = 1 for observed cases and STATUS = 0 for
censored cases. In this study, the survival data are singly
Generally, survival function and hazard function are used right censored so that all the censored cases have a value
to describe the status of customer survival during the of 15 (months) for the variable DUR.
tenure of observation. The survival function gives the
probability of surviving beyond a certain time point t. DATA SOURCES
However, the hazard function describes the risk of event There are four major data sources for this study: block
(in this case, customer churn) in an interval time after time level marketing and financial information, customer level
t, conditional on the customer already survived to time t. demographic data provided through a third party vendor,
Therefore the hazard function is more intuitive to use in customer internal data, and customer contact records. A
survival analysis because it attempts to quantify the brief description of some of the data sources follows.
instantaneous risk that customer churn will take place at
time t given that the customer already survived to time t. Demographic Data – Demographic dada is from a third
party vendor. In this study, the following are examples of
For survival analysis, the best observation plan is customer level demographic information:
prospective. We begin observing a set of customers at - Primary household member’s age
some well-defined point of time (called the origin of time) - Gender and marital status
and then follow them for some substantial period of time, - Number of adults
recording the times at which customer churns occur. It’s - Primary household member’s occupation
not necessary that every customer experience churn - Household estimated income and wealth ranking
(customers who are yet to experience churn are called - Number of children and children’s age
censored cases, while those customers who already - Number of vehicles and vehicle value
churned are called observed cases). Typically, not only - Credit card
do we predict the timing of customer churn, we also want - Frequent traveler
to analyze how time-dependent covariates (e.g. customers - Responder to mail orders
calls to service centers, customers change plan types, - Dwelling and length of residence
customers change billing options, and etc.) impact the
occurrence and timing of customer churn. Customer Internal Data – Customer internal data is from
the company’s data warehouse. It consists of two parts.
SAS/STATâ has two procedures for survival analysis: The first part is about customer information like market
PROC LIFEREG and PROC PHREG. The LIFEREG channel, plan type, bill agency, customer segmentation
procedure produces parametric regression models with code, ownership of the company’s other products, dispute,
censored survival data using maximum likelihood late fee charge, discount, promotion/save promotion,
estimation. The PHREG procedure is a semi-parametric additional lines, toll free services, rewards redemption,
regression analysis using partial likelihood estimation. billing dispute, and so on. The second part of customer
SUGI 27 Data Mining Techniques

internal data is customer’s telecommunications usage data. numerical variables and the kept categorical variables
Examples of customer usage variables are: from the step one.
- Weekly average call counts
- Percentage change of minutes The next step is to use PROC PHREG to further reduce
- Share of domestic/international revenue the number of variables. A stepwise selection method was
used to create a final model with statistically significant
Customer Contact Records – The Company’s Customer effects of 29 exploratory variables on customer churn over
Information System (CIS) stores detailed records of time.
customer contacts. This basically includes customer calls
to service centers and the company’s mail contacts to PROC PHREG DATA = SASOUT2.ALL2 OUTEST =
customers. The customer contact records are then SASOUT2.BETA;
classified into customer contact categories. Among the MODEL DUR*STATUS(0) = &VARLIST1
/ SELECTION = STEPWISE
customer contact categories are customer general inquiry, SLENTRY = 0.0025 SLSTAY = 0.0025 DETAILS;
customer requests to change service, customer inquiry
about cancel, and so on. Model Estimation – With only 29 exploratory variables,
the final data set has reasonable number of variables to
MODELING PROCESS perform survival analysis. Before applying survival
Model process includes the following four major steps. analysis procedures to the final data set, the customer
survival function and hazard function were estimated
Explanatory Data Analysis (EDA) – Explanatory data using the following code. The purpose of estimating
analysis was conducted to prepare the data for the survival customer survival function and customer hazard function
analysis. An univariate frequency analysis was used to is to gain knowledge of customer churn hazard
pinpoint value distributions, missing values and outliers. characteristics. From the shape of hazard function,
Variable transformation was conducted for some customer churn in this study demonstrates a typical hazard
necessary numerical variables to reduce the level of function of a Log-Normal model. As previously
skewness, because transformations are helpful to improve discussed, since the shape of survival distribution and
the fit of a model to the data. Outliers are filtered to hazard function was known, PROC LIFEREG produces
exclude observations, such as outliers or other extreme more efficient estimates (with smaller standard error) than
values that are suggested not to be included in the data PROC PHREG does.
mining analysis. Filtering extreme values from the
training data tends to produce better models because the PROC LIFETEST DATA = SASOUT2.ALL3 OUTSURV =
parameter estimates are more stable. Variables with SASOUT2.OUTSURV
missing values are not a big issue, except for those METHOD = LIFE PLOT = (S, H) WIDTH = 1
demographic variables. The demographic variables with GRAPHICS;
more than 20% of missing values were eliminated. For TIME DUR*STATUS(0);
observations with missing values, one choice is to use RUN;
incomplete observations, but that may lead to ignore
useful information from the variables that have The final step is to estimate customer churn. PROC
nonmissing values. It may also bias the sample since LIFEREG was used to calculate customer survival
observations that have missing values may have other probability. At this step the final data set was divided
things in common as well. Therefore, in this study, 50/50 into two data sets: model data set and validation
missing values were replaced by appropriate methods. data set. The model data set is used to fit the model and
For interval variables, replacement values were calculated the validation data set is used to score the survival
based on the random percentiles of the variable’s probability for each customer. A variable of USE is used
distribution, i.e., values were assigned based on the to distinguish the model data set (set USE = 0) and
probability distribution of the nonmissing observations. validation data set (set USE = 1). In the validation data
Missing values for class variables were replaced with the set, set both DUR and STATUS missing so that cases in
most frequent values (count or mode). the validation data set were not to be used in model
estimation. The following code is to prepare the model
Variable reduction – Started with 212 variables in the data set and validation data set.
original data set, by using PROC FREQ, an initial
univariate analysis of all categorical variables crossed IF RANUNI(0) < 0.5 THEN OUTPUT MODEL;
ELSE OUTPUT VALID;
with customer churn status (STATUS) was carried out to
determine the statistically significant categorical variables DATA SASOUT2._MODEL;
to be included in the next modeling step. All the SET MODEL;
categorical variables with a chi-square value or t statistics USE = 0;
of 0.05 or less were kept. This step reduced the number DATA SASOUT2._VALID;
of variables to 115 (&VARLIST1) – including all the SET VALID;
SUGI 27 Data Mining Techniques

STATUS = .; LP=&XBETA;
DUR = .; T=&TIME;
USE = 1; GAMMA=1/_SCALE_;
ALPHA=EXP(-LP*GAMMA);
DATA SASOUT2.APPEND; PROB=0;
SET SASOUT2._MODEL IF _DIST_='EXPONENT' OR _DIST_='WEIBULL' THEN
SASOUT2._VALID; PROB=EXP(-ALPHA*T**GAMMA);
IF _DIST_='LNORMAL' THEN PROB=1-
PROBNORM((LOG(T)-LP)/_SCALE_);
IF _DIST_='LLOGISTC' THEN
Customer Survival Function
PROB=1/(1+ALPHA*T**GAMMA);
IF _DIST_='GAMMA' THEN DO;
D=_SHAPE1_;
K=1/(D*D);
Survival Probability

Survival U=(T*EXP(-LP))**GAMMA;
PROB=1-PROBGAM(K*U**D,K);
IF D LT 0 THEN PROB=1-PROB;
END;
DROP LP GAMMA ALPHA _DIST_ _SCALE_ _SHAPE1_ D
K U;
RUN;
%MEND PREDICT;
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
DUR (Month) PROC LIFEREG DATA=SASOUT2.APPEND OUTEST=A;
MODEL DUR*STATUS(0) = &VARLIST1 /
Figure 1. Customer Survival Function. DIST=LNORMAL;
OUTPUT OUT=B XBETA=LP CONTROL =USE;
RUN;

%PREDICT(OUTEST=A,OUT=B,XBETA=LP,TIME=03)
Hazard Function
Model Validation – With each customer in the validation
0.06
data set being scored for predicted survival probabilities
0.05 for a specified time (that is, number of months since the
origin of time - August 16, 2000), each customer will have
0.04
P0 through P15 as its predicted survival probabilities. P0 is
Hazard

0.03 Hazard the predicted survival probability at the beginning of

0.02 month 1, which is the origin of time. P1 is the predicted
survival probability at end of month 1; P2 is the predicted
0.01
survival probability at end of month 2, and so on. By
0 sorting customers in ascending predicted survival
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 probabilities for a specified time, if the model works,
DUR (Month) customers with the lowest predicted survival probabilities
will have the highest likelihood to churn for this specified
Figure 2, Customer Hazard Function. time period.

By appending the validation data set to the model data set There are two ways to validate the successfulness of the
like above, the following SAS code is to score each model. One way is to rank the predicted survival
customer’s survival probability in the validation data set. probabilities for a specified time in ascending order into
The PREDICT macro produces predicted survival deciles and then to compare the number of churned
probabilities for each customer at a specified time (for this customer during this specified time period in each decile.
case, number of months after the origin of time), based on For example, we can rank all customers predicted survival
the model fitted by PROC LIFEREG. Refer Allison probabilities P3’s into deciles and then count the number
(1995) for details of this macro. of customers who churned during month 3. The other way
is to put the predicted survival probabilities in the same
%MACRO PREDICT (OUTEST=, order and then compare the number of churned customers
OUT=_LAST_,XBETA=,TIME=); up to this specified time in each decile. An example is to
DATA _PRED_;
rank all customers predicted survival probabilities P3’s
_P_=1;
SET &OUTEST (KEEP=_DIST_ _SCALE_ _SHAPE1_ ) into deciles and then count the number of customers who
POINT=_P_; have churned up to the end month 3 from the origin of
SET &OUT; time.
SUGI 27 Data Mining Techniques

Gains Chart - Customers Who Churned in the

Table 1 describes the number of churners for each decile Third Month
during a specified time period, while Table 2 describes the
number of churners up to a specified time since the origin 120%
of time for each decile. Information about cumulative 100%

Percent of Churners
percentage of churners is also included in the tables. Only
selected time periods are reported in the tables. 80%
Random
60%
Modeled
40%
Table 1, Number of Churners by Decile
During a Specified Month 20%

0%
During Month 1 During Month 3 During Month 6 1 2 3 4 5 6 7 8 9 10 11
De Num Cum. Num Cum. Num Cum.
c. of % of % Of % Decile
Churn Churn Churn
1 441 36.9% 329 34.9% 114 15.9% Figure 3, Model Lift Chart for Identifying Churners
2 269 59.4% 236 59.9% 190 42.5% During the Third Month.
3 179 74.3% 125 73.2% 149 63.3%
4 117 84.1% 90 82.7% 100 77.2%
5 95 92.1% 73 90.5% 85 89.1%
6 55 96.7% 49 95.7% 47 95.7% Gains Chart - Customers Who Churned in the Next
7 23 98.6% 26 98.4% 19 98.3%
6 Months
8 9 99.3% 5 98.9% 7 99.3%
9 5 99.7% 4 99.4% 0 99.3%
10 3 100% 6 100% 5 100% 120%

100%
Percent of Churners 80%
Table 2, Number of Churners by Decile Random
Up to a Specified Month 60%
Modeled
40%
Up to Month 3 Up to Month 6 Up to Month 9
De Num Cum. Num Cum. Num Cum. 20%
c. of % of % Of %
Churn Churn Churn 0%
1 1099 35.2% 1594 30.3% 1889 26.8% 1 2 3 4 5 6 7 8 9 10 11
2 732 58.7% 1302 55.1% 1757 51.8%
3 446 73.0% 834 70.9% 1226 69.2% Decile
4 318 83.2% 578 81.9% 811 80.8%
5 253 91.3% 459 90.6% 623 89.6%
Figure 4, Model Lift Chart for Identifying Churners
6 150 96.1% 273 95.8% 378 95.0%
7 73 98.5% 133 98.4% 200 97.8%
In the Next 6 Months.
8 23 99.2% 47 99.3% 77 98.9%
9 11 99.6% 14 99.5% 29 99.3% CONCLUSION
10 14 100% 25 100% 47 100% This study provides another very powerful statistical tool
– survival analysis to predict customer churn. By ranking
The lift measurement is preferred among marketers for the customers predicted survival probabilities in
evaluating and comparing model performance. At each ascending order, the top two deciles capture 55-60% of
decile it demonstrates the model’s power to beat the churners and the top five deciles capture almost 90% of
random approach or average performance, which is churners. The findings of this study will help
visualized in a lift gain chart. Figure 3 is the cumulative telecommunications companies understand customer
lift gain chart for identifying the customers who churned churn risk and customer churn hazard over the time of
during the third month since the origin of time. From the customer tenure. Overall, this study is helpful in
chart, we see that the top two deciles capture about 60% customizing marketing communications and customer
of churners, while the top five deciles captured about 90% treatment programs to optimally time their marketing
of churners. Figure 4 is the cumulative lift gain chart for intervention efforts.
identifying the customers who have churned in the next 6
months since the origin of time. The top two deciles REFERENCES
capture about 55% of churners, while the top five deciles Allison, Paul D. Survival Analysis Using the SASâ
capture about 90% of churners. System: A Practical Guide, Cary, NC: SAS Institute Inc.
1995. 292pp.

Hosmer, JR. DW, and Lemeshow S. Applied Survival

SUGI 27 Data Mining Techniques

Analysis: Regression Modeling of Time to Event Data.

New York: John Wiley & Sons, 1999.

Lu, Junxiang, Detecting Churn Triggers for Early Life

Customers in the Telecommunications Industry – An
Applications of Interactive Tree Training, Proceedings of
the 2nd Data Mining Conference of DiaMondSUG 2001,
Chicago, IL, 2001.

Rud, Olivia Parr, Data Mining Cookbook, New York:

John Wiley & Sons, 2001.

SAS Institute Inc., SAS/STATâ Users Guide, Version 6,

Forth Edition, Volume 1, Cary, NC: SAS Institute Inc.,
1989. 943pp.

SAS Institute Inc., SAS/STATâ Users Guide, Version 6,

Forth Edition, Volume 2, Cary, NC: SAS Institute Inc.,
1989. 846pp.

SAS Institute Inc., SAS/STATâ Software: Changes and

Enhancements through Release 6.11, Cary, NC: SAS
Institute Inc., 1989. 846pp.

Smith Tyler and Besa Smith, Survival Analysis and the

Application of Cox’s Proportional Hazard Modeling
Using SAS, Proceedings of the 26th Annual SAS Users
Group International Conference, Cary, NC: SAS Institute
Inc., 2001.

ACKNOWLEDGMENTS
I am grateful for the valuable inputs and thoughtful
comments in this study by Robert Klein, group manager of
the modeling development group at Sprint’s Mass
Markets Organization.

I wish to thank Paul Garcia, Sr. director of Database

Marketing and CRM Implementation group at Sprint PCS,
for his encouragement and full support in my modeling
endeavors.

CONTACT INFORMATION
Your comments and questions are valued and encouraged.
Contact the author at:
Junxiang Lu, Ph.D.
Sprint Communications Company
6130 Sprint Pkwy
Overland Park, KS 66251
Work Phone: (913) 762-5621
Fax: (913) 762-0804
Email: [email protected]

SAS and all other SAS Institute Inc. product or service

names are registered trademarks or trademarks of SAS
Institute Inc. in the USA and other countries. ® indicates
USA registration. Other brand and product names are
trademarks of their respective companies.