Linear Models Flashcards by Kyle Webb

Recall the standard linear regression model, define any terms and indices

y_i=B₀+B₁x_i1+B₂x_i2+…+B_kx_ik+e_i

y_i is the ith response

xij is the ith value of the jth regressor

k is the number of regressors (k+1=p parameters)

B₀ is the y-intercept

B_j is coefficient associated with the jth regressor

e_i is an error term, usually assumed iid random with mean 0 and variance sig^2

How well did you know this?

Not at all

Perfectly

Matrix form of regression model and what are any assumptions made (OLS assumptions)?

y = XB + e

E[e]=0

var(e)=sig² I

How well did you know this?

Not at all

Perfectly

OLS estimate of B variance and expected value

Bhat = (X’X)^-1X’Y

Var(Bhat)= sig²(X’X)^-1

E(Bhat)=B

How well did you know this?

Not at all

Perfectly

What does OLS estimate, Bhat, do?

Minimizes the sum of squared residuals (in matrix form: SS_res= [y-yhat]’ [y-yhat])

How well did you know this?

Not at all

Perfectly

SS_res in quadratic form (Y’AY where A is symmetric)

Y’[I - X(X’X)^-1X’]Y

How well did you know this?

Not at all

Perfectly

Global F-test

H₀: B₁=B₂=…=B_k=0

H_a: B_j ≠ 0

Test statistic is MS_reg / MS_res~ F

Where MS_reg = SS_reg / k

SS_reg = SUM (yhat_i - ybar)²

How well did you know this?

Not at all

Perfectly

Write SS_reg in terms of matrix notation

SS_reg = Y’ [X(X’X)^-1X’ - 1(1’1)^-11’]Y

How well did you know this?

Not at all

Perfectly

Describe a cell means model

y_ij= u_i + e_ij

y_ij is the jth observation for the ith group

i=1,2,…,t

j=1,2,…,n_i

u_i is the true mean for the ith group

e_ij is the error term

How well did you know this?

Not at all

Perfectly

For a cell means model, define y, M, and u

How well did you know this?

Not at all

Perfectly

What is the incidence matrix and what is uhat, M’M, (M’M)^-1?

How well did you know this?

Not at all

Perfectly

Less than Full Rank ANOVA Model (overly paramterized)

Where the row length of X is larger than the column length

How well did you know this?

Not at all

Perfectly

Difference between overly parameterized and fully parameterized

Overly paramterized models have columns in the X matrix which are linear combinations of the first column (ie not orthogonal)

How well did you know this?

Not at all

Perfectly

For less than full rank anova model, what happens to Bhat?

Since Bhat= (X’X)^-1X’Y it cannot be estimated because X’X does not have an inverse. But, X’X does have generalized inverses which have to be solved that way

How well did you know this?

Not at all

Perfectly

Consequences of Less than Full rank ANOVA models

Generalized inverses are not unique

There are no unique estimates for B

Serious limitations on what we can estimate and test

Usual focus: contrasts (pairwise comparisons, factorial effects)

Things of critical importance: eigenvalues and rank of matrices (provides degrees of freedom)

How well did you know this?

Not at all

Perfectly

If A and B are nxn square matrices then det[AB]=

det[A]det[B}

How well did you know this?

Not at all

Perfectly

If A is nxn then det[A]=0 iff

A is singular

How well did you know this?

Not at all

Perfectly

The rank of A is…

the greatest number of linearly independent columns (or rows) of A. (Linear dependence implies that at least one column (or row) of A can be written as a linear combination of the other columns (or rows) )

How well did you know this?

Not at all

Perfectly

If A and B are non-singular, then for any matix C

C, AC, CB, ACB

all have the same rank

How well did you know this?

Not at all

Perfectly

If A is an mxn matrix of rank r, then there exist non-singular matrices P and Q such that PAQ is one of the following:

How well did you know this?

Not at all

Perfectly

The rank of AB cannot

exceed the rank of either A or B

How well did you know this?

Not at all

Perfectly

If A is a nxn matrix then det[A]=0 iff

the rank of A is less than n

How well did you know this?

Not at all

Perfectly

The matrix of a quadratic form can always

be chosen to be symmetric

How well did you know this?

Not at all

Perfectly

A and B are said to be congruent matrices iff

there exists a non-singular matrix, C, such that

B=C’AC

We say C is the congruent transformation of A

How well did you know this?

Not at all

Perfectly

Let A be an nxn symmetric matrix of rank r. There exists a non-singular matrix C such that

C’AC=D

where D is a diagonal matrix with exactly r non-sero diagonal elements

How well did you know this?

Not at all

Perfectly

If A and B are congruent matrices, then

they have the same rank

Let C be an mxn matrix with rank r then the ranks of C'C and CC'

are also r

Let A be an nxn matrix. There will always exist

n eigenvalues that satisfy det[A-LI]=0 where L is a diagonal matrix of the n eigenvalues

Let A be an nxn symmetric matrix. The rank of A

is the number of non-sero eigenvalues

Let A be an nxn matrix. A has at least one zero eigenvalue iff

A is singular

Let A be an nxn matrix. The determinant of A is the

product of eigenvalues

Let A be an nxn matrix and let C be any nxn non-singular matrix then A, C^-1AC, CAC^-1

all have the same number of eigenvalues

Let A be an nxn real matrix. A necessary and sufficient condition that there exist a nonzero y that satisfies Ay=ey

is that e be an eigenvalue of A

Let P be an nxn matrix. P is called orthonormal iff

P^-1=P' and therefore PP'=P'P=I

Let A be an nxn matrix, and let P be an nxn orthonormal matrix, then

det[A]=det[P'AP]

Let x and y be nx1 vectors. x and y are called orthogonal if

x'y=0 or y'x=0

Let A be a mxn matrix and B be an nxp matrix. A and B are said to be orthogonal if

AB=0

Let A be nxn symmetric matrix. There exists an orthonormal matrix P such that

P'AP=D where D is a diagonal matrix whose diagonal elements are the eigenvalues of A

Let A be an mxn matrix with rank r\>0. There exist matrices A_L (mxr with rank r) and A_R (rxn with rank r) such that

A=A_LA_R

Define column and row space

Column space of a matrix A is the set of vectors that can be generated as linear combinations of the columns of A. Row space of a matrix A is the set of vectors that can be generated as linear combinations of the rows of A.

Let A be an nxn matrix. A is called positive semidefinite if

a) A=A' (A is symmetric) b) y'Ay ≥ 0 for all y c) There exists at least one y ≠ 0 such that y'Ay=0

If A, an nxn matrix, is positive semidefinite then (3 things)

1) The rank of A is less than n 2) The eigenvalues of A are greater than or equal to 0 3) If P is an nxn non-singular matrix then P'AP is also positive semidefinite

Let A be an nxn matrix. A is positive definite if

a) A=A' (A is symmetric) b) y'Ay \> 0 for all y≠0

If A is positive definite then (3 things)

1) The rank of A is n 2) All of the eigenvalues of A are greater than 0 3) Let P be an nxn non-singular matrix. P'AP is also positive definite

A matrix is called non-negative definite if it is

either positive definite or positive semidefinite

Let C be an mxn matrix with rank r. C'C and CC' are

both non-negative definite. C'C or CC' are positive definite iff they have full rank

Let A be an nxn symmetric non-negative definite matrix. There exists some nxn matrix B such that

B'B=A

Let A and B be nxn symmetric matrices. If A is positive definite, then there exists a non-singular matrix Q such that Q'AQ= ____ and Q'BQ=\_\_\_\_

Q'AQ=I and Q'BQ=D where D is a diagonal matrix whose diagonal elements are the roots of det[B-lambdaA]

If A and B are both non-negative definite, then there exists a matrix Q such that both Q'AQ and Q'BQ are

diagonal

Let A be a nxn non-singular matrix partitioned into A=[A₁₁ A₁₂ A₂₁ A₂₂] where both A₁₁ and A₂₂ are square and non-singular and let A^-1=C What is C?

Let A be an nxnx matrix. We call A idempotent if

AA=A

If A is an nxn idempotent matrix with rank n, then A=

If A is an nxn idempotent matrix of rank less than n, then A is

positive semidefinite

If A is an idempotent matrix with rank r, then it has

r non-zero eigenvalues, each equal to 1

Let A be an nxn (symmetric) idempotent matrix then a) A' is b) Let P be an orthonormal matrix. P'AP is c) Let P be an nxn non-singular matrix. PAP^-1 is d) I-A is

a) (symmetric) idempotent matrix b) (symmetric) idempotent matrix. c) idempotent d) (symmetric) idempotent matrix

The trace of an nxn matrix A is the

sum of the diagonal elements

Let A be an mxn matrix and let B be an nxm matrix. trace[AB]=

trace[BA]

If A, B, C are conformable, then trace[ABC]=

trace[CAB] = trace[BAC] = trace[BCA] = trace[CBA] = trace[ACB]

Let A be an nxn matrix and let P be a non-singular nxn matrix. trace[A] = if P is orthonormal then trace[A]

trace[P^-1AP] trace[P'AP]

Let A be an nxn matrix with eigenvalues lam₁, lam₂, lam₃, ... , lam_n trace[A] =

SUM lam_i

Let A and B be nxn matrices and let a and b be scalars. trace[aA+bB] =

a trace[A] + b trace[B]

Let A be an nxn matrix what does this say about the trace?

trace[A'] = trace[A]

Show the general form to obtain a symmetric matrix

A Moore-Penrose Inverse A⁺ for an mxn matrix A satisfies the following 4 conditions

1) AA⁺ is symmetric 2) A⁺A is symmetric 3) AA⁺A=A 4) A⁺AA⁺=A⁺

Let X be an nxp matrix with rank p. Then X⁺=

(X'X)^-1X'

Let A be an mxn matrix. Then every matrix A has a Moore-Penrose inverse, and \_\_\_\_\_\_

it is unique

(A')⁺=

(A⁺)'

The Moore-Penrose inverse of A⁺ is

If the rank of A is r, then each of the following matrices also has rank r (in terms of Moore-Penroses)

A⁺ AA⁺ A⁺A

If A is non-singular, then A⁺=

A^-1

If A is symmetric idempotent, then A⁺=

The matrices AA⁺, A⁺A, I-AA⁺, I-A⁺A are all

symmetric idempotent

For any matrix A, (A'A)⁺=

A⁺(A')⁺

Let P be an mxm orthonormal matrix. Let Q be an nxn orthonormal matrix, and let A be any mxn matrix. Then (PAQ)⁺=

Q'A⁺P'

Let X=[X₁ X₂] Then XX⁺X₁=

X₁

Let A be an mxn matrix. A^- is called a generalized inverse of A if

AA^-A=A Note, the Moore-Penrose inverse is also a generalized inverse

Let X be an mxn matrix with rank r \> 0. The following conditions hold for the generalized inverse

What can be said about X(X'X)^-X'

It is invariant to the choice of generalized inverse

X(X'X)^-X'=

XX⁺

The following conditions hold for K=X(X'X)^-X'

If A is an nxn non-singular matrix, then the system Ax=y

has a unique solution

The system Ax=y has a solution iff (2 answers)

y is in the column space of A AA^-y=y

Let A be an mxn matrix. The system Ax=0 has a solution other than x=0 iff

the rank of A \< n

Let a and x be nx1 vectors. What are the derivatives with respect to x of a'x, x'a, and x'x

a, a, and 2x

Let A be an mxn matrix of constants and let x be an nx1 vector. Then the derivative of Ax with respect to x is

What is the derivative with respect to x of x'Ax?

2Ax

What is a stationary point, x₀?

x₀ is a solution to df(x)/dx = 0 Could be a min response, max response, or a saddle point

What is the Hessian matrix and what information does it provide?

It is a matrix of second derivatives. It describes the funciton in the neighborhood of the stationary point.

Conditions for Ordinary Least Squares estimation of B

How to decompose variance-covariance matrix (where it is known that the matrix is symmetric)

What does the generalized least squares estimate of B do?

What is the GLS estimate of B?

What is var(a'y) and var(Ay)?

What is E[y'Ay]?

What are possible approaches toward minimizing the variance matrix of Bhat? (sig²(X'X)^-1)

1) min |sig²(X'X)^-1| which minimizes the volume of the confidence ellipsoid of the estimated coefficients 2) min tr[sig²(X'X)^-1] which would minimize the sum of the variances of the estimated coefficients Both put an emphasis on eigenvalues

Steps to show that an estimator is the best, linear, unbiased estimator

Describe the density function for y, a px1 vector from the multivariate normal distribution

If z is a nx1 univariate standard normal random variable then what can be said about z'z?

It is distributed chi-squared with n degrees of freedom

What is the non-centrality parameter?

Three things to consider if trying to show y'Ay follows a chi-squared and how to define non-centrality parameter

What test statistic should be used if Σ is known? What test statistic should be used if σ² is unknown but V is known, where σ²V=Σ?

Chi-squared F-distribution

The classic ANOVA table

Source df MS E[MS] F lambda

X_c'1= var(e)= Σ= Σ^-1=

Define the cell means model

Global F-test for cell means model

ANOVA for cell means global F-test

When using a contrast C for multiple comparison testing, what does the rank of C control?

The type I error rate for the set of comparisons

Describe an identifiable parameter

Describe an estimable funciton

Let A be an mxn matrix. The system Ax=0 has a solution other than x=0 iff

the rank of A

Results of this

All contrasts are \_\_\_\_\_

estimable functions

What are the differences between pairwise comparisons for the cell means model and the effects model?

For the generalized inverse case, test statistic given L'Bhat for an effects model

Where whatever is being tested is in the row space of X (L'=AX)

Reasons the effects model is not very valuable from an analysis perspective

Resquires constraintes to be identifiable Constraints ultimately must be linear combinations of the predicted values Analysis must focus on estimable functions (contrasts are always estimable functions) However, the effects model does generalize very nicely

RBD Model and X matrix

X=[1 T M]