Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Nonlinearity and nonconvexity

  1. Jun 30, 2015 #1
    I am taking a course on linear regression online and it talks about the sum of square difference cost function and one of the points it makes is that the cost function is always convex i.e. it has only one optima.

    Now, reading a bit more it seems that non-linear functions tend to give rise to non-convex functions and I am trying to develop some intuition behind it. So, suppose I take a random model like:

    $$
    f(x) = w_0 x^2 + w_1 \exp(x) + \epsilon
    $$

    And the cost function I choose is the same i.e. I want to minimise:

    $$
    J(w_0, w_1) = (f(x) - w_0 x^2 + w_1 \exp(x))^2
    $$

    What is the intuition that the squared term and the exponential term would give rise to non-convexities?
     
  2. jcsd
  3. Jul 1, 2015 #2

    fzero

    User Avatar
    Science Advisor
    Homework Helper
    Gold Member

    For the example that you've chosen (fixing a missing minus sign),

    $$ \frac{\partial^2 J}{\partial w_0 \partial w_1} = 2 x^2 e^x \geq 0,$$

    is positive semidefinite.

    In a more general case, we will have residuals

    $$ r_i = y_i - \sum_\alpha w_\alpha f_\alpha(x_i)$$

    and we are minimising

    $$ S = \sum_i r_i^2.$$

    The sign of the expressions

    $$ m_{\alpha\beta}= \frac{\partial^2 S}{\partial w_\alpha \partial w_\beta} = 2 \sum_i f_\alpha(x_i) f_\beta(x_i),$$

    determines the convexivity of ##S## as a function of the ##w_\alpha##. When ##\beta = \alpha##, ##m_{\alpha\alpha}## is a sum of squares, so is positive semidefinite. When ##\beta\neq \alpha##, it is possible to find ##m_{\alpha\beta}<0##, depending on the specific functions ##f_\alpha## and the data ##x_i##.
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook




Similar Discussions: Nonlinearity and nonconvexity
  1. Nonlinear regression (Replies: 4)

  2. Nonlinear operators (Replies: 5)

  3. Nonlinear Regression (Replies: 10)

  4. Nonlinear system (Replies: 4)

Loading...