Introduction of Regression

Instructor: Weikang Kao, Ph.D.

 Correlation to Regression

 Recall: Correlation
With correlation our aim is to see if there is any relationship
between x and y.
 How about take a step further?
Can we use one variable to predict another one?
 The Aim of Regression

 In regression, the aim is to use one or more predictors to predict the

outcome variable.
 Simple regression vs multiple regression.
 For example, as fat or sugar intake increases does weight increase?
 Advantages of Regression

 Why regression?
 ANOVA, z and t-tests, and chi-square tests allow us to make comparisons
between multiple groups (categorical predictor levels) means.
 Limitations:
 The biggest limitation is we cannot really include continuous predictors (Age, Weight,
Income, Time, etc.)
 ANOVA does not build a predictive model it simply tells us where differences exist but does
not quantify those differences.
 Regression Line

 From correlation to regression.

 First, the correlation coefficient tells us the strength of relationship between two
variables.
 In bivariate regression, we have a single independent variable (predictor) and a
dependent variable (criterion)
 Regression Line

 The shape of the relationship being MODELED by the correlation coefficient is

linear.
 So, r describes the degree to which a straight line describes the values of the Y
variable across the range of X values.
 Regression line helps us to create the best-fitting line to predict what the values of
the Y variable will be for any given value of X.
Regression
Line example
 Regression Line

 In mathematic, how do we define a straight line?

Y = a + bX
 In regression, the idea is the same.
Y = b0 + b1X
Intercept (b0) and gradient (b1)
 Model of Regression

 Idea of prediction: outcome = (model) + error

 Like Correlation, regression can be positive or negative
 Two important factors in terms of defining a straight line.
 Intercept (b0) and gradient (b1)
 Parameters of Regression

 Those parameters (b0 and b1) are the regression coefficients.

 Regression model:
 Outcome (Y) = [b0 + b1(X1)] + error
 b1 tells us what the model looks like and b0 tells us where the model is.
 These weights are known as coefficients (typically Betas: 𝜷) and coefficients tell
us the relative impact of a predictor (independent variable) on our dependent
variable
 Parameters of Regression

 (Un)Standardized Betas
𝜷 is reserved for standardized predictors while b is used for unstandardized.
b are what we get by default when we use our variables in their raw state. They tell us the unit
change in our dependent variable (in its raw units) for a unit change in our predictor variable
(in its raw units).

Reference: https://www3.nd.edu/~rwilliam/stats1/x92.pdf

With 𝜷 we see the impact of a 1 SD change in the predictor on the dependent.

Variance in
Regression
 Variance in Regression

 Recall: the correlation graph

 The best fitting line is the regression model
 The difference between the regression line and the mean of Y
 The method of least squares
 Helps to find the best fit line, we try to minimized the residual.
 Least Squares Criterion

 We try to identify the values of b0 and b1 the produce the best-fitting linear
function.
 So, we use the observed data to identify and create the values of b 0 and b1 that
minimize the distances between the observed values (Y) and the predicted values (Y-
hat)
 In general, we choose the regression coefficient (b1) and the regression intercept
(b0) that define the regression line that minimized the sum of square residuals.
 min Σ[Y-(b0 + b1X)]2 = min Σ(Y- Ŷ)2
 Regression Coefficients

 Below equations help to compute the least square solution:

 b0 = ȳ - b1x̄
 b1 = byx = = rxy
 Predicting Y From X

 Predicted Score (Y-hat):

 By using the best-fitting model, we can take any value of X, substitute it into the
regression equation, and compute the value of Y.
 Observed Score (Y):
 The actual score we find in our data.
 Residual Score (Error):
 The difference between the predicted score and observed score, when using the same
value of X.
 Variance in Regression

 How to determine whether it is a good regression model?

 The best fitting model vs the basic model
 What is the basic model?
 The mean of y
 Sum of square total: SST
 The overall model is :
 SST = SSR + SSE
 Sum of square Regression (Model): SSR
 Sum of square residual (error): SSE
 Variance in Regression

 SST = SSR + SSE

1. (Observed Y – Ymean) = (Predicted Y –Ymean) + (Observed Y – Predicted Y)
2. Variation in Y = Explained by X + Unexplained by X
3. SSTotal = SSRegression + SSError
 What’s the importance of regression?
 Without regression, what’s the best fitting model?
Total Variance
in Regression:
Observed Y – Ymean
=
Variance Can
be explained:
Predicted Y –Ymean
=
Variance Can
Not Be
Explained:
Observed Y – PredictedY
=
 Y –Ŷ Y –ȳ

Ŷ–ȳ

Ŷ=b 0 + b 1 X
Overall Equation:
 Things to Check

 The regression line ALWAYS passes through the point (X, Y)

 The mean of the predicted scores is ALWAYS equal to the mean of the observed
Y score.
Regression: Tests
 Test the Overall Regression Model

 First, we want to know if the model is significant.

 F-Ratio
 For the overall model
 F = (SSM/df) / (SSR/df) = MSM / MSR

 What is this number?

 Test the Individual Parameter

 Second, we want to know if our parameters (b0, b1, b2, etc.) are significant.
 A bad predictor means, a unit change in predictor result no change in the
predicted value of outcome.
 T-test
 The null of t-test is that the b is zero.
 For parameters:
 R-square in Regression

 How do we know if the model is good in terms of how much variability we can
explain?
 R-square

 The value of R-square will be between 0 to 1. Why?
 Regression: Hypothesis

 The null hypothesis:

 There is no relationship between the X variables and the Y variable
 Alternative hypothesis:
 There is a significant relationship between the X variables and the Y variable
 Regression: Assumptions

1. The type of variable

All predictor variables must be quantitative or categorical
The outcome variable be quantitative and continuous.
2. Non- zero variance
The predictors should have some variations.
3. No perfect multicollinearity
There should be no perfect linear relationship between two or more predictors.
4. Predictors are not correlated with extraneous variables
Influence the reliability of our model, think of the third-party issue.
 Regression: Assumptions

5. Heteroscedasticity
Equal residual variances along regression line
6. Independent errors
Errors are not correlated with others
7. Normally distributed errors
The residuals in the model with a mean of zero.
8. Independence
All the values of the outcome (pairs) variable are independent
9. Linearity
The mean value of the outcome variable for each increment of the predictor lie along a straight line
 Regression

 Before data analysis, some issues must be taken care of.

 Outliers.
 Extreme numbers influence coefficient estimates and model fit.
 Shrinkage.
 and adjusted
 FYI:
https://www.investopedia.com/ask/answers/012615/whats-difference-
between-rsquared-and-adjusted-rsquared.asp
 Sample size?
Mostly 15 cases for per predictor.
Pick the Right Tests:
Scenario Practice
 Scenario 1

 A current study is interested in determine the effect of a new drug upon the
number of pictures recalled. Prior research had established a strong correlation
between painting skills and pictures recall. Therefore, individual differences in
painting skills were controlled to produce a more sensitive test of the treatment
effect.
 What test should we use?
 What are the IV and DV?
 Scenario 2

 Suppose we have data on blood pressure levels for males and females and we
wish to see if one sex has higher/lower blood pressure than the other.
 What test should we use?
 What are the IV and DV?
 Scenario 3

 A current study is interested in whether individuals’ weight change can be

predicted by how many soda they drink per day.
 What test should we use?
 What are the IV and DV?
 Scenario 4

 In a current data we collect, we notice that there is a strong correlation between

consumers’ attitudes toward our brand and how many reward bonuses they
receive every year. We would like to do some further analysis to see if we can use
reward bonus receive to predict consumers’ attitudes.
 What test should we use?
 What are the IV and DV?
Regression: Application
 Regression: Application

Use the “(simplerelationships.xlsx) “ data.

 Income – how much a person earns per year
 Percent Income Saved – what percent of income carried over to next year and can
be negative if more money was spent than came in.
 Score on a savings motivation metric – an ordinal scale.
 Regression: Assumptions

Check assumptions:
 The type of variable:
Is it good?
 Non- zero variance
Use graph or look at the variance.
plot(density(data$Income))
plot(density(data$PI))
plot(density(data$MS))
 Regression: Assumptions

 Multicollinearity:
 We have only one predictor here.
 Predictors are not correlated with extraneous variable.
 Heteroscedasticity:
 The variance around the regression line should be equal across values of x.
scatterplot(data$Income, data$PI)
 Regression: Assumptions

 Independent errors:
 We have no issue because we assume a random sample with no repeated
measurement.
 Normally distributed error:
qqnorm(data$PI)
sshapiro.test(model$residuals)
 Independence:
 Linearity:
scatterplot(data$Income, data$PI)
 Regression: Application

 Build the regression model:

model <- lm(PI ~ Income, data = data)
#PI is the outcome variable and Income is the predictor variable
F (1, 97) = 18.37, p < .001, R2 = .16.

Estimate Std. error T-value P-value

Intercept -8.378e-03 2.570e-03 -3.26 0.00154
Income 1.097e-07 2.560e-08 4.29 < .001
 Regression Model

 Based on the result, now we are able to conduct the regression model.
 The expected regression model is: Y = b0 + b1X
 We can replace the values of intercept (b0) and slope (b1) with our own values.
 Our model is:
 Y = (-0.0083) + (.0000001097 )X
 Regression: Summary Write Up

A simple regression model was conducted to predict participants’ money-

saving motivation, based on their annul income. All the regression assumptions were
met, and no further adjustment made. A significant regression equation was found (F
(1, 97) = 18.37, p < .001), with an R2 of .16. Both the intercept (p = .002) and
predictor (p < .001) were statistically significant. The result suggested that, income
predicts and shows that for each dollar increase in income there is a .0000001097
percent increase in savings.
In Class Practice
 In Class Practice

 Example: Can we use either height or weight to predict heart rate?

 Make sure you test the assumption and do a summary write up.
Height: 175, 170, 180, 178, 168, 181, 190, 185, 177, 162
Weight: 60, 70, 75, 80, 69, 78, 82, 84, 72, 53
Heart Rate: 60, 70, 75, 73, 71, 73, 76, 80, 68, 64
Non-Linear Trends
Non-Linear Trends
Many relationships are not
best captured as straight
lines.

For example, the effect of

stress on performance is
known to follow a quadratic
trend (Yerkes-Dodson curve).

What should we do?

 Non-Linear Trends

 Always plot our data.

Using the scatterplot function in R two lines will be drawn (car library needed):
1. The red lines (by default) is a “smoothed” fit (only useful for thinking about
trend)
2. The green line (by default) plots the best fitting linear trend (regression line).
Non-Linear Trends

 It looks like we have a

non-linear trend – higher
values of the predictor
are related with a
quicker increase (steeper
slope) in our dependent
than lower values.
 We also see the linear
model is going to do a
poor job fitting the data.
 Capturing Non-Linear Trends

 In many cases we can use linear regression to capture non-linear trends.

However, you will want to pay close attention to your residuals as frequently you
will get good model fit, but that fit may be deceptive (e.g., predicting before and
after a curve poorly but other points well).
 In the example graph on the preceding slide it appears the predicted slope gets
steeper as the predictor increases (linearly) – a quadratic trend.
 Such trends can be captured by adding a x^2 of the predictor to the model.
 If we had data that had two visible shifts, we would then consider adding higher
order functions (e.g., x^3).
Capturing Non-
Linear Trends

 It should be noted that a

heuristic for determining your
polynomial effect(s) is to count
the shifts (going from up to
down, up to flat, and so on).
 Conducting a Non-Linear Regression Model

 Quadratic:
 = b0 + b1X1 + b2
 Cubic:
 = b0 + b1X1 + b2 + b2
 Lower Order Trends

 When we include higher order trends it is important, we do not remove their lower order
components.
For example, if we wanted to add a quadratic trend to the model, we need to
keep the linear trend in as well even if its not significant.
 What does that mean?
To include a cubic trend we must include the quadratic and linear trends as
well.
 The reason for this is that if we do not include the lower order trends the higher order
trends will capture in part these effects which would bias our estimates.
 A potential issue that will pop up as you include trends beyond linear is multicollinearity
(predictors sharing some of the same predictive variance – Age and Income).
 Non-Linear Trends: Example

 We try to simulate a non-linear dataset (Lionel, 2016)

set.seed(20191007)
x<-seq(0,50,1)
y<-((runif(1,10,20)*x)/(runif(1,0,10)+x))+rnorm(51,0,1)
 Plot our data:
plot(x,y)
 Non-Linear Trends: Example

 From this graph set approximate starting values

 a_start<-8
 b_start<-2*log(2)/a_start
 Build the model:
model<-nls(y~a*exp(-b*x),start=list(a=a_start,b=b_start))
 Find the best fitting line:
lines(x,predict(model),lty=2,col="red",lwd=3)
 1. When should we use regression instead of
ANOVA?
2. Please explain the relationship between
SStotal, SSregression and SSerror.

In Class 3. Please use the following data to build a

regression model and write a summary. IV is
Practice II sugar and DV is calories.
Sugar: 5, 8, 9, 10, 15, 18, 14, 17, 20, 22, 24, 26,
30 ,30, 32
Calories: 20, 30, 60, 70, 100, 95, 70, 83, 103, 112,
130, 80, 95, 130, 112