How to Differentiate an Integral Involving a Probability Density Function?

[EngWiPy]

Hello,

I have this problem

\frac{\partial}{\partial\,x}\int_0^{∞}\log(1+x)\,f_X(x)\,dx,

where x is a random variable, and f_X(x) is its probability density function.

It's been a long time since I encountered a similar problem, and I forgot how to do this. Do we use Leibniz integral rule here?

Thanks

 

[S.G. Janssens]

Strictly speaking, this derivative is zero. You need to use different symbols for the integration variable and the differentiation variable. The way it is now makes it hard to answer your question.

 

[EngWiPy]

Strictly speaking, this derivative is zero. You need to use different symbols for the integration variable and the differentiation variable. The way it is now makes it hard to answer your question.

What do you mean?

 

[S.G. Janssens]

What do you mean?

You integrate with respect to $x$. This yields a number, a constant. The derivative of a constant is zero.

 

[EngWiPy]

OK right it is like this

\frac{\partial}{\partial\,s}\int_0^{∞}\log(1+x\,s)\,f_X(x)\,dx

 

[S.G. Janssens]

Ok, yes, then you can use Leibniz rule, but probably you better use this version or something similar, because your integral is over an unbounded domain. Basically, you can interchange integration and differentiation pending some technical conditions, the most important of those being that you can bound the partial derivative of the integrand w.r.t. $s$ by some integrable function, uniformly in $s$.

 

[EngWiPy]

Ok, yes, then you can use Leibniz rule, but probably you better use this version or something similar, because your integral is over an unbounded domain. Basically, you can interchange integration and differentiation pending some technical conditions, the most important of those being that you can bound the partial derivative of the integrand w.r.t. $s$ by some integrable function, uniformly in $s$.

So basically it becomes

\int_0^{∞}\frac{x\,f_X(x)}{1+x\,s}\,dx

right?

 

[S.G. Janssens]

Yes, but for this to be rigorous you have to prove that there exists a function $g : [0,\infty) \to \mathbb{R}$ such that $g$ is integrable, i.e.
$$
\int_0^{\infty}{|g(x)|\,dx} < \infty
$$
and furthermore it holds that
$$
\left|\frac{x\,f_X(x)}{1+x\,s}\right| \le |g(x)|
$$
for all $x \ge 0$ and for all $s$ that you are considering. If you are a physicist you probably don't care too much, but that will make me cry a little.

 

[S.G. Janssens]

It may not be that hard, though. For example, if $s$ is positive and bounded away from zero, then $x \mapsto \frac{x}{1 + xs}$ is bounded on $[0,\infty)$, uniformly in $s$. Let $C > 0$ be such a bound. Since $f_X$ is a probability density, you can choose $g = C f$.

 

[EngWiPy]

Yes, but for this to be rigorous you have to prove that there exists a function $g : [0,\infty) \to \mathbb{R}$ such that $g$ is integrable, i.e.
$$
\int_0^{\infty}{|g(x)|\,dx} < \infty
$$
and furthermore it holds that
$$
\left|\frac{x\,f_X(x)}{1+x\,s}\right| \le |g(x)|
$$
for all $x \ge 0$ and for all $s$ that you are considering. If you are a physicist you probably don't care too much, but that will make me cry a little.

If we have $\mathbb{E}\{x\}=\bar{x}$, then we can have $g(x)=xf_X(x)\ge\frac{xf_X(x)}{1+x\,s}$ since $x,\,s\ge 0$, right?

 

[S.G. Janssens]

Yes, if $X$ has finite expectation, that works fine. Very nice

 

[EngWiPy]

OK, perfect. Now comes the second question:

This problem comes from a larger problem, which is an optimization problem. The optimization problem states that

\underset{s\ge 0}{\max}\int_0^{∞}\log(1+x\,s)\,f_X(x)\,dx\\\text{s.t}\,\,\, s\le \bar{s}

and some how the solution ends up like this

s=\left(\frac{1}{x_0}-\frac{1}{x}\right)^+

where $x_0$ is a constant related to the Lagrangian multiplier, and $(a)^+=\max(0,a)$. But how?

 

[S.G. Janssens]

I don't think I understand it entirely. What is $\overline{s}$? Is it also an expectation? Also, why is the solution a function of $x$? So far, I was under the impression that $s$ is a numerical parameter.

 

[EngWiPy]

I don't think I understand it entirely. What is $\overline{s}$? Is it also an expectation? Also, why is the solution a function of $x$? So far, I was under the impression that $s$ is a numerical parameter.

$\bar{s}$ is a maximum value. I should've written $s_{\text{max}}$. Basically $s$ in my problem is a power allocated to a communication system, where the $\log(1+x\,s)$ is the instantaneous capacity of the channel given state $x$. The integral of course is the average capacity. So, I need to optimize $s$ such that the average capacity of the channel is maximized, given that there is a maximum power budget. Makes sense now?

 

[S.G. Janssens]

Makes sense now?

Partially. I still don't understand why the solution you presented in post #12 is a function of $x$. Also, I don't understand why you would use Lagrange multipliers when you have a simple inequality constraint $0 \le s \le s_{\text{max}}$. The way I read it now, is that you need to find $s_0$ such that the function
$$
[0,s_{\text{max}}] \ni s \mapsto \int_0^{\infty}{\log(1 + sx)f_X(x)\,dx}
$$
assumes a local or global maximum in $s_0$. But since $f_X$ is non-negative, doesn't this just mean that $s_0 = s_{\text{max}}$? I suppose not, probably I'm misunderstanding something.

 

[EngWiPy]

Partially. I still don't understand why the solution you presented in post #12 is a function of $x$. Also, I don't understand why you would use Lagrange multipliers when you have a simple inequality constraint $0 \le s \le s_{\text{max}}$. The way I read it now, is that you need to find $s_0$ such that the function
$$
[0,s_{\text{max}}] \ni s \mapsto \int_0^{\infty}{\log(1 + sx)f_X(x)\,dx}
$$
assumes a local or global maximum in $s_0$. But since $f_X$ is non-negative, doesn't this just mean that $s_0 = s_{\text{max}}$? I suppose not, probably I'm misunderstanding something.

That is why I asked the question. I thought I did the partial derivative wrongly. For the constraint, actually it depends on $x$. We need the avreage power below a certain maximum budget.That is

\int_0^∞ s(x)f_X(x)\,dx\leq s_{\text{max}}

The transmit power also depends on $x$, i.e., it is $s(x)$. So, the optimization problem becomes:

<br /> \underset{s(x)\ge 0}{\max}\int_0^{∞}\log(1+x\,s(x))\,f_X(x)\,dx\\\text{s.t}\,\,\, \int_0^∞ s(x)f_X(x)\,dx\leq s_{\text{max}}<br />

See the attached file eq. 4 and 5.

 

[S.G. Janssens]

Aha, so you are maximizing over a set of functions, rather than over a numerical parameter. This makes your problem more interesting, but also more difficult. It requires some knowledge of variational methods (i.e. infinite dimensional optimization). A few general remarks on strategy:

• You need to first carefully describe which functions are admissible by specifying the domain of your capacity functional as a subset of an appropriate function space, taking into account the budget constrain. In infinite dimensions it is typically not trivial to actually prove that this set contains a maximizer.
• The ordinary (numerical) derivative changes into a so-called Gâteaux derivative, which is a derivative of a functional (or, more generally, a nonlinear operator) with respect to a function. (Depending on context, sometimes you need the stronger notion of Fréchet derivative.) In particular, just pretending that $s$ is a numerical parameter and applying Leibniz' rule does not work.
• In order to deal with the inequality constraints some infinite dimensional form of the Kuhn-Tucker conditions may be required.

You might want to have a look at https://www.amazon.com/dp/038790915X/?tag=pfamazon01-20 of Zeidler's book on applied nonlinear functional analysis. I didn't study your PDF, but it may be that there you can also find some pointers to relevant literature.

 

[S.G. Janssens]

Did you notice in your PDF link that in (3) there is inequality while in (4) the constraint is an equality?

It seems to me that one argues a priori that any maximizer of the problem with the inequality constraint must actually satisfy this constraint with equality, thereby eliminating the need for Kuhn-Tucker type conditions and making the problem amenable to Lagrange multipliers.

From the application's point of view, is it natural to assume from the beginning that any $S$ is continuous? (The maximizer in (5) is continuous.) Or would it be more natural to start in some space containing discontinuous functions as well?

 

[EngWiPy]

Did you notice in your PDF link that in (3) there is inequality while in (4) the constraint is an equality?

It seems to me that one argues a priori that any maximizer of the problem with the inequality constraint must actually satisfy this constraint with equality, thereby eliminating the need for Kuhn-Tucker type conditions and making the problem amenable to Lagrange multipliers.

From the application's point of view, is it natural to assume from the beginning that any $S$ is continuous? (The maximizer in (5) is continuous.) Or would it be more natural to start in some space containing discontinuous functions as well?

I think $s$ is continuous, which means it takes any value between $0$ and $s_{\text{max}}$. OK, now with equality in the constraint, how do we get the solution? Any hint?

 

[S.G. Janssens]

Any hint?

Yes, see post #17. You may also find sections 3.1 and 3.5 of Cheney's Analysis for Applied Mathematics helpful and a bit more accessible. For equality constraints the theory presented there should be enough.

Start by choosing a suitable function space, so you can define your objective and constraint functionals on the same open subset of that space and apply Lagrange multipliers. A Banach space of integrable functions seems most natural, but its positive cone has empty interior, which may complicate matters somewhat. Since you already know that your maximizer is continuous, you could cheat a little and try to work directly on a space of continuous functions instead.

 

[EngWiPy]

Yes, see post #17. You may also find sections 3.1 and 3.5 of Cheney's Analysis for Applied Mathematics helpful and a bit more accessible. For equality constraints the theory presented there should be enough.

Start by choosing a suitable function space, so you can define your objective and constraint functionals on the same open subset of that space and apply Lagrange multipliers. A Banach space of integrable functions seems most natural, but its positive cone has empty interior, which may complicate matters somewhat. Since you already know that your maximizer is continuous, you could cheat a little and try to work directly on a space of continuous functions instead.

One of the author has a book, and in it she detailed the solution as following:

The Lagrangian function is given by

\mathcal{L}=\int_0^{∞}\log(1+x\,s(x))\,f_X(x)\,dx-\lambda\left(\int_0^∞ s(x)f_X(x)\,dx-s_{\text{max}}\right)

Taking the derivative with respect to $s(x)$ and set to zero yields

\left(\left[\frac{x}{1+s(x)x}\right]-\lambda\right)f_X(x)=0

which by solving for $s(x)$ yields

s(x)=\left(\frac{1}{\lambda}-\frac{1}{x}\right)^+

So, I guess this all I need to know. Thanks

 

[S.G. Janssens]

What I attempted to explain in my last few posts is how you could turn her purely formal manipulations into a proof. It is easy to write down the expression for the Lagrangian but, for starters, it is not even clear on what space $\mathcal{L}$ is defined, let alone whether it is differentiable. These are non-trivial questions that I was trying to discuss, but are now unfortunately left unanswered. Moreover, being a stationary point of the Lagrangian is a necessary yet not a sufficient condition for being an extremum.

So, I guess this all I need to know. Thanks

I think this is all you want to know.

 

[EngWiPy]

What I attempted to explain in my last few posts is how you could turn her purely formal manipulations into a proof. It is easy to write down the expression for the Lagrangian but, for starters, it is not even clear on what space $\mathcal{L}$ is defined, let alone whether it is differentiable. These are non-trivial questions that I was trying to discuss, but are now unfortunately left unanswered. Moreover, being a stationary point of the Lagrangian is a necessary yet not a sufficient condition for being an extremum.I think this is all you want to know.

Right. That is all I want to know. I think for us as engineers we don't dive into abstract math too deep.

 

[S.G. Janssens]

I think for us as engineers we don't dive into abstract math too deep.

That is such a pity, particularly because this is precisely the kind of mathematics that is very much motivated by applications. In any case, I enjoyed thinking about this problem.

 

[EngWiPy]

As a modification to the above, in my problem I have something like this

<br /> \underset{s_k\ge 0}{\max}\,\,\sum_{k=1}^K\int_0^{∞}\log(1+x_k\,s_k)\,f_{X_k}(x_k)\,dx_k\\\text{s.t}\,\,\, \sum_{k=1}^Ky_k\,s_k=Q<br />

where $y_k$ is also a random variable, and $Q$ is some constant.

In this case, the p.d.f of $x_k$ when we derive the Lagrangian function remains in the derivative, not as in the example above. How to handle this in the subsequent analysis?

 

[EngWiPy]

Can we find the integration first, and then find the derivative?