How can a vector be multiplied by itself ?

[ahmadka]

Hi all ... Well I'm reading a paper and this simple least squares equation is not making sense to me .. This is the equation (I tried using latex but it ain't working well for me):

http://img19.imageshack.us/img19/3046/equationk.png

A little background on this equation:

basically this equation is being used for finding a mapping which takes some high dimensionality data (x), and maps it to some low dimensional data (y). The mapping transformation matrix (a) is what we have to find. The mapping should be such that we minimize the error between the original and projected data (as you can probably tell) ..

Inside the summation, y and x are being accessed serially in a column by column manner, and so y_i and x_i are referring to individual columns of matrices y and x, respectively .. Also, as you can probably tell, both y and x contain the same number of columns, m. Only the number of rows they each have is different ..

Now here's my question .. As far as I can tell (and I'm virtually sure I'm correct here), the result of the evaluation being performed inside the brackets would result in a column vector ... And then this column vector is being squared !? That is, this is what the equation is suggesting:

( <column vector here> )²

How is this possible ? Or is the equation maybe referring to a dot product here ?

 

[micromass]

Doesn't a^Tx_i result in a 1x1-matrix? I.e. a number

 

[ahmadka]

Doesn't a^Tx_i result in a 1x1-matrix? I.e. a number

Nope ... a^T is a matrix with both dimensions being greater than 1 .. x_i and y_i are column vectors ...

so this is basically what is happening here:

((matrix*column vector) - column vector) = (column vector - column vector) = column vector

(column vector)² ... !??!

 

[micromass]

They probably mean the inproduct with itself. I.e.

$$(x_1,...,x_n)^2=x_1^2+...+x_n^2$$

I don't like the notation however...

 

[D H]

Doesn't a^Tx_i result in a 1x1-matrix? I.e. a number

Exactly.

The way to read this expression is that a and each x_i are row vectors -- i.e., 1×N matrices. This makes a^T an N×1 matrix, which in turn makes a^Tx_i a scalar.

 

[Landau]

I agree with DH. Also see here.

 

[ahmadka]

The way to read this expression is that a and each x_i are row vectors -- i.e., 1×N matrices. This makes a^T an N×1 matrix, which in turn makes a^Tx_i a scalar.

The thing is with regards to the problem at hand, it makes more sense for a to be a matrix, and not a vector. I say this because a is a transformational matrix which takes a high dimensional column vector and turns it into a low dimensional column vector .. So for example, suppose x contains m column vectors which are each n dimensions long (so then x would be an nxm matrix), and we want to reduce the number of dimensions these column vectors use, from n to p let's say, and so n>p .. Let x_i refer to anyone of the m column vectors in x, then ...

y_i = a^Tx_i

where y_i contains the reduced number of dimensions, i.e. p ..

So for such a problem, a cannot be column vector .. rather is would be a pxn transformation matrix which transforms a nx1 column vector into a px1 column vector ...

So I really don't see how for the problem at hand, a can be a column vector ..

I agree with DH. Also see here.

Thats me again .. :)

 

[D H]

The thing is with regards to the problem at hand, it makes more sense for a to be a matrix, and not a vector. ...

Ask yourself: Does that interpretation make any sense? Look at the equation to which you linked in the original post:

$$\mathbf a = \underset{\mathbf a}{\operatorname{arg\,max}} \sum_{i=1}^m \left(\mathbf a^T\mathbf x_i-y_i\right)^2$$

This means that a^Tx must have the same dimensionality as y_i. Since y_i is written in non-bold text, one can assume that each y_i is a scalar. This means that each x_i must be a n-vector (a row vector), as must a.

Backing up a bit, suppose you are doing an experiment where you are need to assess how a bunch of different independent variables affect the outcome of the experiment, with the outcome expressed as a scalar measurable quantity. To keep things simple, your first guess is that this scalar quantity is some unknown linear function of the independent variables. One approach to finding this linear relation is to find the solution that minimizes the sum of the squares of the residuals. That is exactly what the above equation does.