Equation related to Linear Discriminant Analysis (LDA)?

  • Thread starter zak100
  • Start date
  • #1
462
11

Homework Statement


I cant understand an equation to LDA. The context is:
The objective of LDA is to perform dimensionality reduction while
preserving as much of the class discriminatory information as
possible
Maybe the lecturer is trying to create a proof of the equation given below.

I know the above that LDA projects the points along an axis so that we can have maximum separation between two classes.in addition to reducing dimesionality

Homework Equations


I am not able to understand the following equation:
##Y =W^T## ##X##

It says that:
Assume we have a set of D-dimensional samples ##{x_1, x_2,...x_N},## ##N_1## of belong to class
##\Omega_1## and ##N_2## to class ##\Omega_2##. We seek to obtain a scalar ##Y## by projecting the samples ##X## onto a line:
In the above there is no W. So I want to know what is W?

The Attempt at a Solution



W might represent the projection line? But T = transpose.

Somebody please guide me. For complete description, please see the attached file.

Zulfi.
[/B]
 

Attachments

Answers and Replies

  • #2
StoneTemplePython
Science Advisor
Gold Member
1,180
581
If you are patient enough, we can step through this.

There are some severe notation and definitional roadblocks that will come up. From past threads, you know that a projection matrix satisfies ##P^2 = P## i.e. idempotence implies it is square and in fact diagonalizable and in fact full rank iff it is the identity matrix, yet this contradicts slide 8 of your attachment. (I have a guess as to what's actually being said here, but the attachment is problematic. My guess btw is that ##W^T W = I## but ## WW^T = P##)

Typically more than half the battle is clearly stating what is being asked, then I'd finish it off with something a bit esoteric like matrix calculus or majorization. The fact mentioned on page 9 that LDA can be interpreted / derived as a Max Likelihood method for certain normals... is probably the most direct method.
 
Last edited:
  • #3
462
11
Hi,
Thanks for your reply? Do you mean that W is the sample matrix?

Zulfi.
 
  • #4
StoneTemplePython
Science Advisor
Gold Member
1,180
581
e_
Hi,
Thanks for your reply? Do you mean that W is the sample matrix?

Zulfi.
Have you looked at pages 7 and 9 in Detail? It seems fairly clear to me that ##W## is made up. Equivalently, you choose it, and you should choose optimally (page 9).

- - - -
My belief, btw, is that page 8 shows

##J(W) = \frac{\det\big(W^T S_b W\big)}{\det\big(W^T S_W W\big)}##

where ##S_W## and ##S_B## are symmetric positive (semi?) definite matrices. However since I've conjectured that ##W^TW = I## but ##WW^T=P## my belief is you select ##W## to be a rank ##r## matrix and hence

##J(W) = \frac{\det\big(W^T S_b W\big)}{\det\big(W^T S_W W\big)} = \frac{e_r\big(W^T S_b W\big)}{e_r\big(W^T S_b W\big)}= \frac{e_r\big(P S_b \big)}{e_r\big(P S_b \big)}= \frac{e_r\big(P S_b P\big)}{e_r\big(P S_b P\big)}##

where ##e_r## is the rth elementary symmetric function of the eigenvalues of the matrix inside. But these notes clearly are part of a much bigger sequence and are not standalone. There should be a notational lookup somewhere.
 
Last edited:
  • #5
462
11
Hi,
Thanks. You mean that W represents the Matrix of Eigen Vectors.

Kindly tell me what is the difference between ##\mu## and ##\hat{\mu}## in slide #3. ##\mu## represents the mean of X values where as ##\hat{\mu}## represents the mean of Y values. If both are mean why we use ^ symbol with one and other one is without ^ symbol. We could have represented them using ##\mu_1## and ##\mu_2##. I can't understand this.

Please guide me.

Zulfi.
 

Related Threads on Equation related to Linear Discriminant Analysis (LDA)?

  • Last Post
Replies
2
Views
2K
Replies
2
Views
725
  • Last Post
Replies
5
Views
469
  • Last Post
Replies
3
Views
1K
Replies
21
Views
3K
Replies
1
Views
2K
Replies
1
Views
2K
Replies
2
Views
2K
Replies
17
Views
2K
Top