Solving Minimization Problem Involving Variance & Covariance

Click For Summary
SUMMARY

The discussion centers on minimizing the expected squared deviation, specifically addressing the problem of finding the constant \(b\) that minimizes \(E[(X-b)^2]\). It is established that the optimal value is \(b=E[X]\), confirmed through differentiation of the expected value expression. The second part of the discussion involves minimizing \(E[(Y-aX-b)^2]\), where \(Y\) is a random variable, leading to a constrained minimization problem involving variances and covariances. The key takeaway is the distinction between constants and random variables in optimization contexts.

PREREQUISITES
  • Understanding of expected value and variance in probability theory
  • Familiarity with covariance and its implications in statistical analysis
  • Basic knowledge of optimization techniques, particularly in calculus
  • Concept of constrained optimization in statistics
NEXT STEPS
  • Study the properties of covariance and its role in regression analysis
  • Learn about constrained optimization techniques in statistics
  • Explore the application of Lagrange multipliers in optimization problems
  • Investigate the implications of random variables in statistical modeling
USEFUL FOR

Statisticians, data analysts, and anyone involved in statistical modeling and optimization, particularly in the context of variance and covariance analysis.

OhMyMarkov
Messages
81
Reaction score
0
Hello Everyone!

What $b$ minimizes $E[(X-b)^2]$ where $b$ is some constant, isn't it $b=E[X]$? Is it right to go about the proof as follows:

$E[(X-b)^2] = E[(X^2+b^2-2bX)] = E[X^2] + E[b^2]-2bE[X]$, but $E = b$, we differentiate with respect to $b$ and set to zero, we obtain that $b=E[X]$. Is this proof correct? I was thinking it was until I got this problem:

What $Y$ minimizes $E[(Y-aX-b)^2]$? The given expression contains variances and covariances, but all I get was $Y=aE[X]+b$.

What am I doing wrong here?

Any help is appreciated! :D
 
Physics news on Phys.org
OhMyMarkov said:
Hello Everyone!

What $b$ minimizes $E[(X-b)^2]$ where $b$ is some constant, isn't it $b=E[X]$? Is it right to go about the proof as follows:

$E[(X-b)^2] = E[(X^2+b^2-2bX)] = E[X^2] + E[b^2]-2bE[X]$, but $E = b$, we differentiate with respect to $b$ and set to zero, we obtain that $b=E[X]$. Is this proof correct?


Correct


I was thinking it was until I got this problem:

What $Y$ minimizes $E[(Y-aX-b)^2]$? The given expression contains variances and covariances, but all I get was $Y=aE[X]+b$.

What am I doing wrong here?

Any help is appreciated! :D

The problem with this second question is that with normal naming conventions \(Y\) is a random variable not a constant, if it were a constant what you get would be correct. If it is a RV then it leaves you with a minimisation problem where the variables are \( \overline{Y}\), \(Var(Y)\), \( Covar(Y,aX+b)\). This is a constrained minimisation problem as \( | Covar(Y,aX+b)| \le \sqrt{Var(Y)Var(aV+b)}\)

CB
 
Last edited:
Thank you CaptainBlack!

But, you mentioned [FONT=MathJax_Math-italic]E[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main]][FONT=MathJax_Main],[FONT=MathJax_Math-italic]V[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]r[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main]][FONT=MathJax_Main],[FONT=MathJax_Math-italic]C[FONT=MathJax_Math-italic]o[FONT=MathJax_Math-italic]v[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main],[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]+[FONT=MathJax_Math-italic]b[FONT=MathJax_Main]] , what about [FONT=MathJax_Math-italic]E[FONT=MathJax_Main][[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]][FONT=MathJax_Main],[FONT=MathJax_Math-italic]V[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]r[FONT=MathJax_Main][[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]] , can we sub them for [FONT=MathJax_Math-italic]C[FONT=MathJax_Math-italic]o[FONT=MathJax_Math-italic]v[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main],[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]+[FONT=MathJax_Math-italic]b[FONT=MathJax_Main]] ? Or are the variables intentionally used in this fashion so that the hand calculation becomes easier?
 
Last edited:
OhMyMarkov said:
Thank you CaptainBlack!

But, you mentioned [FONT=MathJax_Math-italic]E[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main]][FONT=MathJax_Main],[FONT=MathJax_Math-italic]V[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]r[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main]][FONT=MathJax_Main],[FONT=MathJax_Math-italic]C[FONT=MathJax_Math-italic]o[FONT=MathJax_Math-italic]v[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main],[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]+[FONT=MathJax_Math-italic]b[FONT=MathJax_Main]] , what about [FONT=MathJax_Math-italic]E[FONT=MathJax_Main][[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]][FONT=MathJax_Main],[FONT=MathJax_Math-italic]V[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]r[FONT=MathJax_Main][[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]] , can we sub them for [FONT=MathJax_Math-italic]C[FONT=MathJax_Math-italic]o[FONT=MathJax_Math-italic]v[FONT=MathJax_Main][[FONT=MathJax_Math-italic]Y[FONT=MathJax_Main],[FONT=MathJax_Math-italic]a[FONT=MathJax_Math-italic]X[FONT=MathJax_Main]+[FONT=MathJax_Math-italic]b[FONT=MathJax_Main]] ? Or are the variables intentionally used in this fashion so that the hand calculation becomes easier?

\(E(X)\) and \(Var(X)\) are constants for this problem, while \(u=Covar(Y,aX+b)\) is one of the variable in the optimisation problem.

CB
 

Similar threads

MHB Distribution, expected value, variance, covariance and correlation
  • · Replies 14 ·
Replies
14
Views
2K
MHB How Do You Minimize Mean Square Deviation in Dice Roll Predictions?
  • · Replies 4 ·
Replies
4
Views
1K
Undergrad On Jensen's inequality for conditional expectation
  • · Replies 1 ·
Replies
1
Views
2K
Graduate Computing a variance in astrophysics context
  • · Replies 1 ·
Replies
1
Views
1K
MHB Cdf, expectation, and variance of a random continuous variable
  • · Replies 1 ·
Replies
1
Views
2K
Undergrad Help understanding conditional expectation identity
  • · Replies 1 ·
Replies
1
Views
2K
MHB No problem! Happy to help. (Smile)
  • · Replies 9 ·
Replies
9
Views
2K
Undergrad Expected value of X~Geom(p) given X+Y=z
  • · Replies 5 ·
Replies
5
Views
2K
MHB Counterfactual Expectation Calculation
  • · Replies 1 ·
Replies
1
Views
2K
MHB Calc Expected Value & Variance of Multivar. Func.
  • · Replies 1 ·
Replies
1
Views
2K