Data Mining For Business Intelligence: Shmueli, Patel & Bruce
Data Mining For Business Intelligence: Shmueli, Patel & Bruce
Data Mining For Business Intelligence: Shmueli, Patel & Bruce
Data Reduction
Data Exploration Visualization
Supervised Learning
Goal: Predict a single target or outcome
variable
Training data, where target value is known Score to data where value is not known
Unsupervised Learning
Goal: Segment data into meaningful segments;
detect patterns
There is no target (outcome) variable to predict or
classify
Methods: Association rules, data reduction &
exploration, visualization
Supervised: Classification
Goal: Predict categorical target (outcome)
variable Examples: Purchase/no purchase, fraud/no fraud, creditworthy/not creditworthy Each row is a case (customer, tax return, applicant) Each column is a variable Target variable is often binary (yes/no)
Supervised: Prediction
Goal: Predict numerical target (outcome) variable
applicant) Each column is a variable Taken together, classification and prediction constitute predictive analytics
what Example: If X was purchased, Y was also purchased Rows are transactions Used in recommender systems Our records show you bought X, you may also like Y Also called affinity analysis
simpler/smaller data Reducing the number of variables/columns (e.g., principal components) Reducing the number of records/rows (e.g., clustering)
Data Exploration
Data sets are typically large, complex & messy
Need to review the data to help refine the task Use techniques of Reduction and Visualization
2.
3. 4.
5.
6. 7.
8.
9.
Define/understand purpose Obtain data (may involve random sampling) Explore, clean, pre-process data Reduce the data; if supervised DM, partition it Specify task (classification, clustering, etc.) Choose the techniques (regression, CART, neural networks, etc.) Iterative implementation and tuning Assess results compare models Deploy best model
sample from a database, to produce statisticallyvalid results XLMiner, e.g., limits the training partition to 10,000 records Once you develop and select a final model, you use it to score the observations in the larger database
effectively train a model A popular solution: oversample the rare cases to obtain a more balanced training set Later, need to adjust results for the oversampling
Pre-processing Data
Types of Variables
Determine the types of pre-processing
Variable handling
Numeric
Most algorithms in XLMiner can handle numeric
Detecting Outliers
An outlier is an observation that is extreme,
being distant from the rest of the data (definition of distant is deliberately vague) Outliers can have disproportionate influence on models (a problem if it is spurious) An important step in data pre-processing is detecting outliers Once detected, domain knowledge is required to determine if it is an error, or truly extreme.
Detecting Outliers
In some contexts, finding outliers is the purpose
omit them If many records are missing values on a small set of variables, can drop those variables (or use proxies) If many records have missing values, omission is not practical
Solution 2: Imputation Replace missing values with reasonable substitutes Lets you keep the record and use the rest of its (nonmissing) information
largest scales would dominate and skew results Puts all variables on same scale Normalizing function: Subtract mean and divide by standard deviation (used in XLMiner) Alternative function: scale to 0-1 by subtracting minimum and dividing by the range
Useful when the data contain dummies and numeric
explanations of relationships between variables The fit may be excellent When used with new data, models of great complexity do not do so well.
1000 Revenue
800
600
400
200
0 0 100 200 300 400 500 600 700 800 900 1000
Expenditure
Overfitting (cont.)
Causes:
Too many predictors A model with too many parameters Trying many different models
Consequence: Deployed model will not work as well as expected with completely new data.
model Validation partition to implement the model and evaluate its performance on new data
Test Partition
When a model is developed on
training data, it can overfit the training data (hence need to assess on validation) Assessing multiple models on same validation data can overfit validation data Some methods use the validation data to choose a parameter. This too can lead to overfitting the validation data Solution: final selected model is applied to a test partition to give unbiased estimate of its performance on new
per capita crime rate by town proportion of residential land zoned for lots over 25,000 sq.ft. proportion of non-retail business acres per town. Charles River dummy variable (1 if tract bounds river; 0 otherwise) nitric oxides concentration (parts per 10 million) average number of rooms per dwelling proportion of owner-occupied units built prior to 1940 weighted distances to five Boston employment centres index of accessibility to radial highways full-value property-tax rate per $10,000 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town Median value of owner-occupied homes in $1000
Specifying Output
24 -6.246905549 33.4 4.783477282 36.2 8.435659135 28.7 3.079596801 16.5 4.954169128 18.9 -0.235661871 18.9 -3.05655773 23.1 2.299458015 17.5 0.553144385
Summary of errors
Training Data scoring - Summary Report
Total sum of squared errors 6977.106
RMS error
Error = actual - predicted RMS = Root-mean-squared error = Square root of average squared error
In previous example, sizes of training and validation sets differ, so only RMS Error and Average Error are comparable
models can be handled within the modest limits of Excel and XLMiner Models can then be used to score larger databases XLMiner has functions for interacting with various databases (taking samples from a database, and scoring a database from a developed model)
Summary
Data Mining consists of supervised methods
(Classification & Prediction) and unsupervised methods (Association Rules, Data Reduction, Data Exploration & Visualization) Before algorithms can be applied, data must be characterized and pre-processed To evaluate performance and to avoid overfitting, data partitioning is used Data mining methods are usually applied to a sample from a large database, and then the best model is used to score the entire database