Linear Transformations if the design matrix

[Mark53]

Homework Statement

given that X is an n × p matrix with linearly independent columns.

And $$X^∗ = XA$$ where A is an invertible p × p matrix.

a)

Show that: $$X^*{({X^*}^TX^*)^-}^1{X^*}^T = X{(X^TX)^-}^1X^T$$

b)
Consider two alternative models
$$M : Y = Xβ + ε$$ and $$M^∗ : Y = X^∗β ^∗ + ε$$

Show that $$η^∗ = η$$, i.e., the vector of fitted values is the same, whatever the form of the design matrix X.

The Attempt at a Solution

a)$$X^*{({X^*}^TX^*)^-}^1{X^*}^T = XA{(X^TA^TXA)^-}^1X^TA^T$$

multiplying by A inverse

$$=X{(X^TA^TXA)^-}^1X^TA^T$$

can I then multiply it by the A transpose to give:

$$=X{(X^TA^TXA)^-}^1X^TA^TA^T$$

$$=X{(X^TA^TXA)^-}^1X^TA$$

Then the inverse of A again to get

$$=X{(X^TA^TXA)^-}^1X^T$$

not sure if this is the right process

b)

not sure how to get started on this part

 

[StoneTemplePython]

I gathered all scalars are in reals
- - - -

For starters: if
$X^∗ = XA$

then

$\big(X^∗\big)^T \neq X^TA^T$

do you get why $H = X{(X^TX)^-}^1X^T$ exists, and why $H$ is a projector? This is sometimes called the Hat Matrix and it's worth spending some time drilling it.

We can worry about part (b) later. I don't think you defined what $\beta^*$ is, or fully stated the definition of $\eta$ just yet.

 

[Mark53]

I gathered all scalars are in reals
- - - -

For starters: if
$X^∗ = XA$

then

$\big(X^∗\big)^T \neq X^TA^T$

do you get why $H = X{(X^TX)^-}^1X^T$ exists, and why $H$ is a projector? This is sometimes called the Hat Matrix and it's worth spending some time drilling it.

We can worry about part (b) later. I don't think you defined what $\beta^*$ is, or fully stated the definition of $\eta$ just yet.

Wait would it be

$$(X^*)^T = (XA)^T= A^TX^T$$

which would give$$XA{(A^TX^TXA)^-}^1A^TX^T$$

Where would I go from here?

 

[StoneTemplePython]

Wait would it be

$$(X^*)^T = (XA)^T= A^TX^T$$

which would give$$XA{(A^TX^TXA)^-}^1A^TX^T$$

Where would I go from here?

Yes... now do a dimensions check...

is $A$ invertible? is $X$?

 

[Mark53]

Yes... now do a dimensions check...

is $A$ invertible? is $X$?

We know that A is invertible which means we can:

$$XA{A^-}^1{(A^TX^TXA)^-}^1A^TX^T$$

$$=X{(A^TX^TXA)^-}^1A^TX^T$$

 

[StoneTemplePython]

I have no idea how you went from

$XA{(A^TX^TXA)^-}^1A^TX^T$

to

We know that A is invertible which means we can:

$$XA{A^-}^1{(A^TX^TXA)^-}^1A^TX^T$$

In general matrix multiplication is not commutative... I'm getting the sense that we're hitting a fundamental gap in your understanding of matrix algebra.

What I was hinting at was, for 3 invertible matrices $Q, R, S$

$\big(QRS\big)^{-1}= S^{-1}R^{-1}Q^{-1}$

Using associativity of matrix multiplication you should be able to find 3 invertible matrices to apply that to, and go from there. Forum rules prohibit me giving the complete answer.

To be honest, if you're stumbling on transposing and inverting matrices, this problem may be out of reach. If this is for a course in statistics, the math will get more involved as you progress... I'd suggest drilling the linear algebra heavily over the next few weeks to get there. There are a lot of good free resources out there like this: https://math.byu.edu/~klkuttle/0000ElemLinearalgebratoprint.pdf , which has lots of problems and solutions to selected exercises.

 

[Mark53]

I have no idea how you went from

$XA{(A^TX^TXA)^-}^1A^TX^T$

to
In general matrix multiplication is not commutative... I'm getting the sense that we're hitting a fundamental gap in your understanding of matrix algebra.

What I was hinting at was, for 3 invertible matrices $Q, R, S$

$\big(QRS\big)^{-1}= S^{-1}R^{-1}Q^{-1}$

Using associativity of matrix multiplication you should be able to find 3 invertible matrices to apply that to, and go from there. Forum rules prohibit me giving the complete answer.

To be honest, if you're stumbling on transposing and inverting matrices, this problem may be out of reach. If this is for a course in statistics, the math will get more involved as you progress... I'd suggest drilling the linear algebra heavily over the next few weeks to get there. There are a lot of good free resources out there like this: https://math.byu.edu/~klkuttle/0000ElemLinearalgebratoprint.pdf , which has lots of problems and solutions to selected exercises.

$XA{(A^TX^TXA)^-}^1A^TX^T$

$=XA{A^-}^1{X^-}^1{(X^T)^-}^1{(A^T)^-}^1A^TX^T$
$=XI{X^-}^1{(X^T)^-}^1({A^-}^1A)^TX^T$
$=X{(X^TX)^-}^1IX^T$
$=X{(X^TX)^-}^1X^T$

Is this correct now?

 

[StoneTemplePython]

$XA{(A^TX^TXA)^-}^1A^TX^T$

$=XA{A^-}^1{X^-}^1{(X^T)^-}^1{(A^T)^-}^1A^TX^T$
$=XI{X^-}^1{(X^T)^-}^1({A^-}^1A)^TX^T$
$=X{(X^TX)^-}^1IX^T$
$=X{(X^TX)^-}^1X^T$

Is this correct now?

getting closer. The issue which I alluded to with "dimensions check" is that $X$ is not square so it cannot have an actual inverse... It is square after being multiplied by a certain other matrix which use or parenthesis (associativity) can alleviate.

 

[Mark53]

getting closer. The issue which I alluded to with "dimensions check" is that $X$ is not square so it cannot have an actual inverse... It is square after being multiplied by a certain other matrix which use or parenthesis (associativity) can alleviate.

$X^TX$

That would be a square matrix, which means it has an inverse