Scribd Logo
Upload

Welcome to Scribd!

  • 16 views

    A Short Introduction To Linear Mixed Models

    Uploaded by

    marikum74

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    A Short Introduction To Linear Mixed Models

    Uploaded by

    marikum74
    16 views5 pages

    Original Title

    mixed_models

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    16 views5 pages

    A Short Introduction To Linear Mixed Models

    Uploaded by

    marikum74

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    You are on page 1of 5

    A short introduction to Linear Mixed Models

    Xiang Ao
    October 5, 2007

    1 Linear Mixed Models


    1.1 Random Intercept Model
    Let’s start with an example. Students learn in classes; classes are taught within
    school; schools are administered within school districts. A typical multilevel
    model (or mixed model; or sometimes called hierarchical models) would assign
    students to level 1, class level 2, school level 3 and district level 4. Ignoring the
    clustering structure of your data set will result in biased or inconsistent estima-
    tion of the coefficients and often downward-biased standard error estimation.
    Let’s first consider a two-level situation: students at level 1 and schools at
    level 2. Suppose our model for students’ test scores is:

    yij = αj + rij

    where
    rij ∼ N (0, σ 2 )
    Here i is student; j is school index. We express a student’s score in terms of
    the sum of an intercept for school (αj ) and a random error rij associated with
    student i at school j.
    Now school level intercepts can be expressed as the sum of an overall mean
    µ and random deviations from that mean (uj ):

    αj = µ + uj

    where
    uj ∼ N (0, σu2 ).
    Putting the two equations together we have a multilevel model:

    yij = µ + uj + rij

    where
    rij ∼ N (0, σ 2 )
    and
    uj ∼ N (0, σu2 ).

    1
    We can see that without the extra term uj in the model, we basically have
    a simple mean model; or say we have a linear regression model with only a
    constant term as the regressor. By introducing ui into the model, we are able
    to take account of the school effect in this case. That is, we model not only
    the grand mean, but also the mean within school. Also, we have the benefit of
    decomposing the total variance into two components: σ 2 and σu2 . This way we
    can explain how much of the variation comes from variation between schools
    and how much comes from variation within schools (between students). To
    be able to do these, we sacrifice by making an extra distribution assumption
    uj ∼ N (0, σu2 ).
    Let’s consider another example. Suppose we have patent prior art data. We
    have the proportion of prior art by examiners as the dependent variables. We
    have three levels: patent applicants, examiners, and technology. These three
    levels are crossed; they are not nested as students, schools, districts. Our model
    would be:

    yij = µ + uj + vk + wl + rijkl
    where
    rijkl ∼ N (0, σ 2 )
    and
    uj ∼ N (0, σu2 ),
    vk ∼ N (0, σv2 ),
    2
    wl ∼ N (0, σw ).

    1.2 Random Intercept and Random Slopes Model


    We discussed about about the random intercept models; that is models which
    only allows the intercept to vary across groups. Now let’s introduce the model
    with slopes varying across groups (levels).
    Suppose we have predictors like student’s family income in our test scores
    example:
    yij = αj + βj xij + rij
    rij ∼ N (0, σ 2 )

    αj = µ + uj
    βj = η + vj
    where
       2 
    uj σ σuv
    ∼ N (0, u )
    vj σuv σv2
    In this model, we allow both the intercept and slopes of predictors vary
    across groups (levels). We make the assumption that the random components

    2
    of the intercept and slopes are jointly distributed as multivariate normal. σu2
    is the variance of level 2 residuals uj from predicting level 1 intercept, σv2 is
    the variance of level 2 residuals vj from predicting level 1 slope. σuv is the
    covariance between uj and vj .

    1.3 Variance Decomposition: ANOVA vs Mixed Models


    Let’s consider a simple model of intercept-only two-level model. One-way ANOVA
    and mixed model can be represented by the same model:

    yij = µ + uj + rij
    The difference in these two methods is: ANOVA is basically putting in group
    (level) dummies in the regression and mixed model is putting group-varying
    components into the error term.
    ANOVA calculates the intraclass correlation by calculating between-group
    mean square error and within-group mean square error. Intraclass correlation is
    ratio of between-group variance and total variance (the sum of between-group
    variance and within-group variance). A linear mixed model treat ui as part of
    the variance of the model; then it estimates the variance-covariance matrix.
    For a model with only one group variable, the difference between variance
    decomposition by ANOVA and mixed model may not be huge. However, for a
    model with multiple group variables, such as

    yij = µ + uj + vk + wl + rijkl .
    If the three group variables are not orthogonal, then the variance decompo-
    sition by ANOVA depends on the order they are put in the model, while in a
    mixed model, the order is totally irrelevant.

    1.4 General Linear Mixed Models


    In general, a linear mixed model (LMM) is defined by

    y = Xβ + Zγ + e
    where
    e ∼ N (0, R)
    γ ∼ N (0, G)
    , and e and γ are uncorrelated.
    This model has two parts: Xβ is the fixed effect part, and Zγ is the random
    effect part.
    In this model, E(y) = Xβ and Var(y) = Var(Zγ) + Var(e) = ZGZ0 + R.
    This means that in the linear mixed models, the unconditional mean is
    determined by the fixed effect part only.
    The other way to formulate this is:

    3
    y|γ ∼ N(Xβ + Zγ, R)
    where
    γ ∼ N (0, G)
    .

    2 Generalized Linear Mixed Models


    Generalized Linear Mixed Models (GLMM) are used to model non-normal data
    or normal data with correlations or heteroskadasticities. Generalized Linear
    Models (GLM) deal with data with distributions that belong to exponential
    family, such as Logit, Poisson. GLMM is an expansion of GLM to incorporate
    normally distributed random effects.
    We know GLM usually models the mean of the distribution: g(E[yi ])) = x0i β;
    that is, some function g() of the mean of the distribution is a linear function of
    xi ’s. This function g() is called link function. In the case of logit, it is the logit
    function. In the case of Poisson model, it is the log function.
    Now the difference between GLM and GLMM is that GLMM has an extra
    term in it:
    g(E[yi | γ])) = x0i β + z 0 γ
    where γ is normally distributed with mean 0 and variance G.
    In Linear Mixed Models, we have E[yi | γ] = E[yi ]. With a linear model,
    E[yi | γ] = x0i β + z 0 E[γ] = x0i β since γ ∼ N (0, G). With a non-linear model like
    GLMM,
    E[yi | γ]) = g −1 (x0i β + z 0 γ),
    where z 0 γ remains since g −1 is usually a non-linear function.
    The other way to formulate this is by two steps:
    First, the fixed and random effects are combined to form a linear predictor

    η = Xβ + Zγ.
    Second, if we have a linear mixed model, then we simply need to add in a
    normally distributed error term:

    y = η + e = Xβ + Zγ + e.
    Equivalently, we have

    y|γ ∼ N(Xβ + Zγ, R)


    where
    γ ∼ N (0, G)
    .

    4
    Now if we don’t have y normally distributed, then we have to “link” the in-
    dependent variable to the linear predictor η, therefore the name “link function”.

    g(y) = η

    We have

    y|γ ∼ N(g−1 (η), R).


    It says that the conditional distribution of y given η is a normal distribution,
    with mean as a function of the linear combination of fixed and random effects.
    For example, say we have a log link (for example, in a Poisson model we
    have a log link), then

    E[y] = E[exp(X0 β + Z0 γ)] = exp(X0 β) E[exp(Z0 γ)],


    which means the unconditional expectation of y is not only a function of fixed
    effects, but also random effects.

    3 Estimate the models


    In general, GLMM is estimated in the maximum likelihood framework,
    Z Y
    L= fyi |γ (yi |γ)fγ (γ)dγ,
    i

    Basically, this says that we need to integrate out the random effect part
    to get the unconditional likelihood function. Unfortunately usually we don’t
    have an analytic solution for this, since random effects can be high-dimensional.
    Three possible solutions are proposed:

    • Penalized Quasi-Likelihood. SAS Glimmix uses this approach. This ap-


    proach uses a second order approximation of the log density, based on
    Taylor expansion. Essentially it determines the marginal log likelihood
    as that of an approximate linear mixed models. This approach lacks of a
    “true” log likelihood, but computationally feasible in many occasions.

    • Integration by Gauss-Hermite Quadrature. This is essentially doing nu-


    merical integration to estimate the true likelihood function. This approach
    has the highest precision but often not feasible if there are many random
    effects, since it’s very computationally intensive. SAS NLMIXED, Stata’s
    xtnlmixed uses this approach.
    • Laplace approximation. This approach is relatively fast, but not as accu-
    rate. R’s lme4 package uses it currently for GLMM’s.

    You might also like