Ridge Regression Minimization Proof

[zzmanzz]

Homework Statement

Linear family:

f(x;a) = a_{o} + (a_{1}.a_{2},a_{3},...,a_{k}) \cdot x[\tex]<br /> <br /> (Xa - Y)^t \sigma^{-1} (Xa-Y) + \lambda (a^t a-a^2_{o} [\tex]&lt;br /&gt; &lt;br /&gt; a = (X^t \sigma^{-1} X + \lamda I_{o})^{-1} x^t \sigma^{-1} Y [\tex]&amp;lt;br /&amp;gt; &amp;lt;br /&amp;gt; &amp;lt;h2&amp;gt;Homework Equations&amp;lt;/h2&amp;gt;Y_{i} = f(x_{i}) + N_{i} [\tex]&amp;amp;lt;br /&amp;amp;gt; &amp;amp;lt;br /&amp;amp;gt; i = 1,2,3,4,...,k.&amp;amp;lt;br /&amp;amp;gt; &amp;amp;lt;br /&amp;amp;gt; given data \{ (x_{i},y_{i})\}^2_{i} [\tex]&amp;amp;amp;lt;br /&amp;amp;amp;gt; &amp;amp;amp;lt;br /&amp;amp;amp;gt; x_{i} [\tex] = vector&amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;gt; y_{i} [\tex] = scalar&amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; to minimize we write &amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; (X(a+\Delta a) - Y)^t \sigma^{-1} (X(a+\Delta a)-Y)[\tex]&amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;gt; =(Xa - Y)^t \sigma^{-1} (Xa-Y) + \Delta a^t X^t \sigma^{-1} (Xa-Y)+(Xa-Y)^t \sigma^{-1} + O(\Delta a^t \Delta a)[\tex]&amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;gt; =(Xa - Y)^t \sigma^{-1} (Xa-Y) + 2\Delta a^t X^t \sigma^{-1} (Xa-Y) + O(\Delta a^t \Delta a)&amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; [\tex]&amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; cond for a:&amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; X^t \sigma^{-1} Xa - X^t \sigma^{-1} X = 0[\tex]&amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; a = (X^t \sigma^{-1} X)^(-1)X^t \sigma^{-1} Y[\tex]&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;h2&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;The Attempt at a Solution&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;/h2&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; Sorry can someone tell me why my latex is off? I am new to the forums and did my best to use the code? Thanks, I will repost.

 

[Ray Vickson]

Homework Statement

Linear family:

f(x;a) = a_{o} + (a_{1}.a_{2},a_{3},...,a_{k}) \cdot x[\tex]<br /> <br /> (Xa - Y)^t \sigma^{-1} (Xa-Y) + \lambda (a^t a-a^2_{o} [\tex]&lt;br /&gt; &lt;br /&gt; a = (X^t \sigma^{-1} X + \lamda I_{o})^{-1} x^t \sigma^{-1} Y [\tex]&amp;lt;br /&amp;gt; &amp;lt;br /&amp;gt; &amp;lt;h2&amp;gt;Homework Equations&amp;lt;/h2&amp;gt;&amp;lt;br /&amp;gt; &amp;lt;br /&amp;gt; &amp;lt;br /&amp;gt; Y_{i} = f(x_{i}) + N_{i} [\tex]&amp;amp;lt;br /&amp;amp;gt; &amp;amp;lt;br /&amp;amp;gt; i = 1,2,3,4,...,k.&amp;amp;lt;br /&amp;amp;gt; &amp;amp;lt;br /&amp;amp;gt; given data \{ (x_{i},y_{i})\}^2_{i} [\tex]&amp;amp;amp;lt;br /&amp;amp;amp;gt; &amp;amp;amp;lt;br /&amp;amp;amp;gt; x_{i} [\tex] = vector&amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;gt; y_{i} [\tex] = scalar&amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; to minimize we write &amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;gt; (X(a+\Delta a) - Y)^t \sigma^{-1} (X(a+\Delta a)-Y)[\tex]&amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;gt; =(Xa - Y)^t \sigma^{-1} (Xa-Y) + \Delta a^t X^t \sigma^{-1} (Xa-Y)+(Xa-Y)^t \sigma^{-1} + O(\Delta a^t \Delta a)[\tex]&amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;gt; =(Xa - Y)^t \sigma^{-1} (Xa-Y) + 2\Delta a^t X^t \sigma^{-1} (Xa-Y) + O(\Delta a^t \Delta a)&amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; [\tex]&amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; cond for a:&amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;gt; X^t \sigma^{-1} Xa - X^t \sigma^{-1} X = 0[\tex]&amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; a = (X^t \sigma^{-1} X)^(-1)X^t \sigma^{-1} Y[\tex]&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;h2&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;The Attempt at a Solution&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;/h2&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; Sorry can someone tell me why my latex is off? I am new to the forums and did my best to use the code? Thanks, I will repost.

&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;\&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot; starts a TeX/LaTeX command; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;/&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot; signals &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;end of TeX. That is, use &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot; instead of &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;[\tex]&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;.&amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; RGV