Cauchy Schwarz equality implies parallel

  • Thread starter Bipolarity
  • Start date
  • #1
775
1
I'm learning about Support Vector Machines and would like to recap on some basic linear algebra. More specifically, I'm trying to prove the following, which I'm pretty sure is true:
Let ##v1## and ##v2## be two vectors in an inner product space over ##\mathbb{C}##.
Suppose that ## \langle v1 , v2 \rangle = ||v1|| \cdot ||v2|| ##, i.e. the special case of Cauchy Schwarz when it is an equality. Then prove that ##v1## is a scalar multiple of ##v2##, assuming neither vector is ##0##.

I've tried using the triangle inequality and some other random stuff to no avail. I believe there's some algebra trick involved, could someone help me out? I really want to prove this and get on with my machine learning.

Thanks!

BiP
 

Answers and Replies

  • #3
775
1
Proving this should not require the definition of the inner product, only the properties.
 
  • #5
775
1
Conjugate symmetry, linearity in the first argument, and positive-definiteness.
 
  • #6
13,802
10,948
Looks to me as another version of the cosine formula if applied to v1+v2
 
  • #7
179
2
By definition, [itex]\langle v_1, v_2 \rangle = \| v_1 \| \cdot \| v_2 \| \cdot \cos(\theta)[/itex] where [itex]\theta[/itex] is the angle between vectors [itex]v_1[/itex] and [itex]v_2[/itex]. If you also additionally know that [itex] \langle v_1, v_2 \rangle = \| v_1 \| \cdot \| v_2 \| [/itex], then the angle between the two vectors must either be 0 or 180 degrees. So they are parallel; hence one is a scalar multiple of the other.
 
  • #8
399
117
That's the definition? It would be true in a real inner product space, but this one is over ℂ.
 
  • Like
Likes rs1n
  • #9
179
2
That's the definition? It would be true in a real inner product space, but this one is over ℂ.
You are absolutely right! My eyes failed me, somehow.
 
  • #10
PeroK
Science Advisor
Homework Helper
Insights Author
Gold Member
15,027
7,204
I'm learning about Support Vector Machines and would like to recap on some basic linear algebra. More specifically, I'm trying to prove the following, which I'm pretty sure is true:
Let ##v1## and ##v2## be two vectors in an inner product space over ##\mathbb{C}##.
Suppose that ## \langle v1 , v2 \rangle = ||v1|| \cdot ||v2|| ##, i.e. the special case of Cauchy Schwarz when it is an equality. Then prove that ##v1## is a scalar multiple of ##v2##, assuming neither vector is ##0##.

I've tried using the triangle inequality and some other random stuff to no avail. I believe there's some algebra trick involved, could someone help me out? I really want to prove this and get on with my machine learning.

Thanks!

BiP
One way to do it is to consider the vector ##u = v_2 - \frac{<v1, v2>}{<v1, v1>} v_1##

Look at ##<u, u>## and show that it's zero when you have C-S equality. This also leads to a proof of the C-S inequality.
 
  • #11
179
2
To get back to the problem, though... over the complex numbers, the inner product is presumably a Hermitian inner product. So

##\begin{align*}
\| u + v \|^2 & = \langle u + v, u+v \rangle = \langle u,u \rangle + \langle u,v \rangle + \langle v,u \rangle + \langle v, v \rangle\\
& = \langle u,u \rangle + \langle u,v \rangle + \overline{\langle u,v \rangle} + \langle v, v \rangle \\
& = \langle u,u \rangle + 2 \mathrm{Re}(\langle u,v \rangle) + \langle v, v \rangle\\
& = \| u\|^2 + 2 \mathrm{Re}(\langle u,v \rangle) + \| v\|^2
\end{align*}##

Similarly,

## 0 \le \| u + \lambda v \|^2 = \| u\|^2 + 2 \mathrm{Re}(\overline{\lambda} \langle u,v \rangle) + |\lambda|^2 \| v\|^2##

Let $$\lambda = -\frac{\langle u, v\rangle }{\|v \|^2}$$ and the right hand side (above) will simplify to the C.S. inequality. Equality occurs if $$\| u + \lambda v \| = 0$$
 
  • #12
177
61
There are few possible ways of doing that. The first one is just to follow the proof of the Cauchy--Schwarz. Namely, for real ##t## consider $$\|v_1 - t v_2\|^2 = \|v_1\|^2 +t^2\|v_2\|^2 - 2t (v_1, v_2) = \|v_1\|^2 +t^2\|v_2\|^2 - 2t \|v_1\|\cdot \|v_2\| = (\|v_1\|-t\|v_2\|)^2.$$ The right hand side of this chain of equations is ##0## when ##t=\|v_1\|/\|v_2\|##. So for this ##t## you get that ##v_1-tv_2=0##, which is exactly what you need.

Another way is more geometric and probably more intuitive. You define ##w## to be the orthogonal projection of ##v_2## onto the one dimensional subspace spanned by ##v_1##, ##w= \|v_1\|^{-2} (v_2, v_1) v_1##. Then ##(v_1, v_2)= (v_1, w)## (checked by direct calculation) and ##v_2-w## is orthogonal to ##v_1## (and so to ##w##).
Therefore ##\|v_2\|^2 =\|w\|^2+\|v_2-w\|^2##.

By Cauchy--Schwarz ## (v_1, w) \le \|v_1\|\cdot \|w\|##, but on the other hand ##(v_1, w) = (v_1, v2) = \|v_1\|\cdot \|v_2\|##, so ##\|v_1\|\cdot \|v_2\| \le \|v_1\|\cdot \|w\|## and therefore ##\|v_2\|\le \|w\|##. Comparing this with ##\|v_2\|^2 =\|w\|^2+\|v_2-w\|^2## we conclude that ##v_2-w=0##.

The second proof is a bit longer, but it is more intuitive, in a sense that it is a pretty standard reasoning used when one works with orthogonal projections.
 
  • #13
PeroK
Science Advisor
Homework Helper
Insights Author
Gold Member
15,027
7,204
Another way is more geometric and probably more intuitive. You define ##w## to be the orthogonal projection of ##v_2## onto the one dimensional subspace spanned by ##v_1##, ##w= \|v_1\|^{-2} (v_2, v_1) v_1##. Then ##(v_1, v_2)= (v_1, w)## (checked by direct calculation) and ##v_2-w## is orthogonal to ##v_1## (and so to ##w##).
Therefore ##\|v_2\|^2 =\|w\|^2+\|v_2-w\|^2##.

By Cauchy--Schwarz ## (v_1, w) \le \|v_1\|\cdot \|w\|##, but on the other hand ##(v_1, w) = (v_1, v2) = \|v_1\|\cdot \|v_2\|##, so ##\|v_1\|\cdot \|v_2\| \le \|v_1\|\cdot \|w\|## and therefore ##\|v_2\|\le \|w\|##. Comparing this with ##\|v_2\|^2 =\|w\|^2+\|v_2-w\|^2## we conclude that ##v_2-w=0##.

The second proof is a bit longer, but it is more intuitive, in a sense that it is a pretty standard reasoning used when one works with orthogonal projections.
The second method is what I suggested in post #10. And, in fact, you can prove Cauchy Schwartz more intuitively this way.
 
  • #14
775
1
I see! Thank you all for your replies! I knew I had seen it somewhere, little did I know it was right there in the proof of the C-S inequality itself!

BiP
 

Related Threads on Cauchy Schwarz equality implies parallel

  • Last Post
Replies
4
Views
2K
  • Last Post
Replies
8
Views
2K
  • Last Post
Replies
8
Views
3K
Replies
6
Views
999
  • Last Post
Replies
5
Views
8K
Replies
8
Views
864
Replies
2
Views
688
Replies
3
Views
2K
  • Last Post
Replies
1
Views
612
  • Last Post
Replies
6
Views
4K
Top