Neural Nets in AI: What's the Intelligent Part?

[Silviu]

Hello! I learned today about neural/deep nets, convolutional nets and things related to this. I am not sure I understand why is this considered AI. The ideas behind are really beautiful, but in the end there are just better (faster) ways of taking derivatives, which anyone can do once given the rules (and in this case a huge amount of time). So beside the obvious increase in speed, I can't see any intelligent output from the computer. Can someone explain to me what is the intelligent part of neural nets? Thank you!

 

[anorlunda]

You have to take the label AI with a grain of salt. There are no language police to enforce who can and can not use the term to mean anything they want.

That goes double for the free-standing word intelligence. There is no universal agreement what that means, including your meaning when you use the word or my meaning.

 

[Silviu]

You have to take the label AI with a grain of salt. There are no language police to enforce who can and can not use the term to mean anything they want.

That goes double for the free-standing word intelligence. There is no universal agreement what that means, including your meaning when you use the word or my meaning.

So what is the most generally accepted definition of AI then? I don't have in my mind a personal definition, but following mechanically some pre-establish rules (which neural nets seem to do, if I understand them right) doesn't sound at all as intelligence to me...

 

[cosmik debris]

So what is the most generally accepted definition of AI then? I don't have in my mind a personal definition, but following mechanically some pre-establish rules (which neural nets seem to do, if I understand them right) doesn't sound at all as intelligence to me...

I think the AI part of neural nets is the learning ability. What sounds like intelligence to you?

Cheers

 

[Drakkith]

So what is the most generally accepted definition of AI then? I don't have in my mind a personal definition, but following mechanically some pre-establish rules (which neural nets seem to do, if I understand them right) doesn't sound at all as intelligence to me...

How's that different from our own intelligence?

 

[phyzguy]

Hello! I learned today about neural/deep nets, convolutional nets and things related to this. I am not sure I understand why is this considered AI. The ideas behind are really beautiful, but in the end there are just better (faster) ways of taking derivatives, which anyone can do once given the rules (and in this case a huge amount of time). So beside the obvious increase in speed, I can't see any intelligent output from the computer. Can someone explain to me what is the intelligent part of neural nets? Thank you!

Take a look at this thread. A deep neural network that was only programmed with the rules of chess was then turned loose to play against itself and learn from the results of the games it played. In a matter of 24 hours, it could beat the best chess playing programs, which can already beat the best humans. If this is not intelligence, it is at least getting close. Nobody programmed it with knowledge of the best moves or the best strategies, it learned them from experience.

 

[jedishrfu]

I'd watch this video by 3blue1brown to get a good understand of how a neural-net works and what it can and can't do:

 

[atyy]

As anorlunda said above in post #2, "intelligence" is a poorly defined term.

Recurrent neural nets can be equivalent to Turing machines: https://doi.org/10.1006/jcss.1995.1013.

 

[Silviu]

How's that different from our own intelligence?

Well for example if you show the 10 digits to a person on a computer (that has never seen them before), he would be able to recognize handwritten ones quite accurately (I actually tried this with some chinese characters). So we are able to extrapolate using even one or 2 examples. Neural nets (based on my understanding) use millions of examples to find a function that minimize the errors in guessing the right answer. Giving it 1-2 example would not allow it to find the right function. So in my perspective it is not guessing the right image/digit/letter etc. using any sort of intelligence, it just takes some derivatives that applies later to similar cases i.e. if u train it to the 10 digits and u give to it something else, it won't be able to say it is something else (it would most probably say it is one of the digits with high probability) while a human would be able to say that something is new.

 

[Silviu]

As anorlunda said above in post #2, "intelligence" is a poorly defined term.

Recurrent neural nets can be equivalent to Turing machines: https://doi.org/10.1006/jcss.1995.1013.

I totally agree with that, and I am not saying I know how to define it. But despite that, it is hard for me to accept that following a prewritten (by a human) algorithm which does nothing more than taking derivatives, can be considered as intelligence (by any definition you would choose).

 

[phyzguy]

I totally agree with that, and I am not saying I know how to define it. But despite that, it is hard for me to accept that following a prewritten (by a human) algorithm which does nothing more than taking derivatives, can be considered as intelligence (by any definition you would choose).

You're missing the point on neural nets. There is no "prewritten (by a human) algorithm". You really need to read about how they work.

 

[donpacino]

Well for example if you show the 10 digits to a person on a computer (that has never seen them before), he would be able to recognize handwritten ones quite accurately (I actually tried this with some chinese characters)

Who did you try this with? A child who has never seen any form of text? or perhaps an adult who has had exposure to other forms of written text and has experience in what to look for.

Lets take this as an example, how does a human child learn a stove is hot and will hurt a hand if placed on the stove. All that happens is they get too close to a heat source and an ion channel opens up, causing a nerve fiber to activate. This sends a signal to the brain saying "hurt" and the brain signals that there is pain, so the body knows to remove the hand to prevent damage. The brain then realizes that at the time and place its body was when the hurt signal was received, its hand was on the stove. Would you call it intelligence to not put a hand on a stove again?

i.e. if u train it to the 10 digits and u give to it something else, it won't be able to say it is something else (it would most probably say it is one of the digits with high probability) while a human would be able to say that something is new.

That is not true. There are systems out there that can recognize when a new signal/pattern is presented.

If I showed you a new symbol without context, you would have NO way of knowing what it was or how it was used. With context, you could make estimated guesses of how it was used. in your head youll compare what is likely to be true, and what is not likely to be true. An AI would do the same thing. The difference is we can't see in your brain to see what process it is using.

 

[phyzguy]

Well for example if you show the 10 digits to a person on a computer (that has never seen them before), he would be able to recognize handwritten ones quite accurately (I actually tried this with some chinese characters). So we are able to extrapolate using even one or 2 examples. Neural nets (based on my understanding) use millions of examples to find a function that minimize the errors in guessing the right answer. Giving it 1-2 example would not allow it to find the right function. So in my perspective it is not guessing the right image/digit/letter etc. using any sort of intelligence, it just takes some derivatives that applies later to similar cases i.e. if u train it to the 10 digits and u give to it something else, it won't be able to say it is something else (it would most probably say it is one of the digits with high probability) while a human would be able to say that something is new.

How many times have you seen digits while your neural net was being "trained" while you were growing up? Probably not millions of times, but I bet at least many thousands. So when you are extrapolating based on the 1-2 examples, you are taking advantage of years of training. Neural nets basically do the same thing.

 

[atyy]

I totally agree with that, and I am not saying I know how to define it. But despite that, it is hard for me to accept that following a prewritten (by a human) algorithm which does nothing more than taking derivatives, can be considered as intelligence (by any definition you would choose).

Yes, there are open problems like how neural networks can perform symbolic manipulation, variable binding etc. There are interesting discussions in:
https://arxiv.org/abs/1711.08378
https://arxiv.org/abs/1410.5401
https://www.nature.com/articles/nature20101
https://www.researchgate.net/publication/309089885_Artificial_intelligence_Deep_neural_reasoning
https://deepmind.com/blog/differentiable-neural-computers/
https://deepmind.com/blog/neural-approach-relational-reasoning/

I also like the discussion of what "intelligence" is in https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec.

 

[FactChecker]

Hello! I learned today about neural/deep nets, convolutional nets and things related to this. I am not sure I understand why is this considered AI. The ideas behind are really beautiful, but in the end there are just better (faster) ways of taking derivatives, which anyone can do once given the rules (and in this case a huge amount of time). So beside the obvious increase in speed, I can't see any intelligent output from the computer. Can someone explain to me what is the intelligent part of neural nets? Thank you!

Clearly, what can be learned in one day is not enough to fully appreciate those subjects. The algorithms should not be compared to "taking derivatives". The ability for neural networks to recognize patterns, define categories, and apply those patterns and categories to new examples is fundamental. If that's not a large part of artificial intelligence, I don't know what is.

 

[BWV]

Neural networks can be viewed as sophisticated nonlinear regression models. Simple NNs reduce to a linear regression if you remove the activation function. Just as a linear regression can approximate any linear function, a NN can approximate any nonlinear one. The training or learning is just an optimization algorithm to minimize the error function. With a linear regression this function has a closed form and is simply the covariance of the squared errors divided by the variance. With NNs you need more sophisticated optimization algorithms

 

[phyzguy]

Neural networks can be viewed as sophisticated nonlinear regression models. Simple NNs reduce to a linear regression if you remove the activation function. Just as a linear regression can approximate any linear function, a NN can approximate any nonlinear one. The training or learning is just an optimization algorithm to minimize the error function. With a linear regression this function has a closed form and is simply the covariance of the squared errors divided by the variance. With NNs you need more sophisticated optimization algorithms

Interesting. Two questions:
(1) Could the same thing be said about the neural networks in our brains, or do you believe that "our" neural networks are somehow qualitatively different from artificial neural networks?

(2) What about NNs that incorporate some stochasticity? In this case can it still be viewed as a nonlinear regression?

 

[atyy]

Interesting. Two questions:
(1) Could the same thing be said about the neural networks in our brains, or do you believe that "our" neural networks are somehow qualitatively different from artificial neural networks?

The learning rule in the brain is generally considered to be different from the backpropagation algorithm.

However, in some cases, biologically plausible learning rules can be seen to be implementing an approximate stochastic gradient descent, eg. see the appendix of Legenstein et al, A Reward-Modulated Hebbian Learning Rule Can Explain Experimentally Observed Network Reorganization in a Brain Control Task, Journal of Neuroscience, 2010.

Famously, the dopamine neurons in our brain also fire in a way that resembles the temporal difference error of some reinforcement learning algorithms, eg. Schultz, Dayan and Montague, A Neural Substrate of Prediction and Reward, Science, 1997.

(2) What about NNs that incorporate some stochasticity? In this case can it still be viewed as a nonlinear regression?

Baldi and Sadowski, The Dropout Learning Algorithm, 2015
Ge et al, Escaping From Saddle Points, 2017

 

[newjerseyrunner]

I totally agree with that, and I am not saying I know how to define it. But despite that, it is hard for me to accept that following a prewritten (by a human) algorithm which does nothing more than taking derivatives, can be considered as intelligence (by any definition you would choose).

Why? It's an algorithm that performs pattern recognition. And using back-propagation is not the only way to train a neural network. Genetic and reinforcement algorithms are also used and are anything but human designed. I've used these genetic algorithms to breed AIs for all sorts of things. Using techniques like that you can have AI so powerful that it'll learn and master DOOM by just playing it over and over again.

 

[ADDA]

A more efficient and simpler to comprehend is Expectation Maximization:

public class EM {
    public class CONSTANTS {
     
        public static final int NFRAMES_COUNT = 3;
        public static final int NFRAMES_SETS = 2;
     
        public static final float THRESH = 1.0f/ 6.0f;
    }

    private float[] theta;
    private float[][] ml;
    private boolean[][] inputOn, inputNeutral, inputOff;
    private int n, nsets, ncount;
 
//default constructor used
//interfaced a create function (if needed by other files)
    public void create() {
        this.n = 3; // this number is somewhat redundant.
        this.ncount = CONSTANTS.NFRAMES_COUNT;
        this.nsets = CONSTANTS.NFRAMES_SETS;
 
        this.theta = new float[this.n];
        this.ml = new float[this.nsets][this.n];

        //three states of input; must match ncount, n
        this.inputOn = new boolean[this.nsets][this.ncount];
        this.inputNeutral = new boolean[this.nsets][this.ncount];
        this.inputOff = new boolean[this.nsets][this.ncount];
     
        //initialize all theta (probabilities) to lower bound
        this.theta[0] = 0.001f;
        this.theta[1] = 0.001f;
        this.theta[2] = 0.001f;
    }
 
//boolean input length (ncount * nsets)
    public void setInput(boolean[] notesOn, boolean[] notesNeutral, boolean[] notesOff) {
        int itr, aitr, bitr;
     
        bitr = 0;
        for (itr = 0; itr < this.nsets; itr += 1) {
            for (aitr = 0; aitr < this.ncount; aitr += 1) {
                this.inputOn[itr][aitr] = notesOn[bitr];
                this.inputNeutral[itr][aitr] = notesNeutral[bitr];
                this.inputOff[itr][aitr] = notesOff[bitr];
             
                bitr += 1;
            }
        }
     
 
        return;
    }
 
    private void EStep() {
        float mult;
        int itr, a,b,c, aitr;
     
        for (aitr = 0; aitr < this.nsets; aitr += 1) {
//always count; somewhat less efficent; could be place in create() function
            a = 0;
            for (itr = 0; itr < this.ncount; itr += 1) {
                if (this.inputOff[aitr][itr]) a += 1;
            }
            b = 0;
            for (itr = 0; itr < this.ncount; itr += 1) {
                if (this.inputNeutral[aitr][itr]) b += 1;
            }
            c = 0;
            for (itr = 0; itr < this.ncount; itr += 1) {
                if (this.inputOn[aitr][itr]) c += 1;
            }
         

//this and not that (my own modus ponens addition); typically just this; could edit to (2 * this.ncount) to clamp the multiplier to 1.0f
            mult = ((float)((a + (this.ncount - (b + c)))) / (float)(this.ncount));
            if (mult < 0.01f) mult = 0.01f;
         
            this.ml[aitr][0] = this.theta[0] * mult;
         
            mult = ((float)((b + (this.ncount - (a + c)))) / (float)(this.ncount));
            if (mult < 0.01f) mult = 0.01f;
         
            this.ml[aitr][1] = this.theta[1] * mult;
         
            mult = ((float)((c + (this.ncount - (b + a)))) / (float)(this.ncount));
            if (mult < 0.01f) mult = 0.01f;
         
            this.ml[aitr][2] = this.theta[2] * mult;
        }
  
        return;
    }

 
    private void MStep() {
        float[][] ntheta = new float[this.nsets][this.n];
        int itr, c, aitr;
 
        for (aitr = 0; aitr < this.nsets; aitr += 1) {
            c = 0;
            for (itr = 0; itr < this.ncount; itr += 1) {
                if (this.inputOff[aitr][itr]) c += 1;
            }
            ntheta[aitr][0] = (float)c * this.ml[aitr][0] + 0.001f;
         
            c = 0;
            for (itr = 0; itr < this.ncount; itr += 1) {
                if (this.inputNeutral[aitr][itr]) c += 1;
            }
            ntheta[aitr][1] = (float)c * this.ml[aitr][1] + 0.001f;
         
            c = 0;
            for (itr = 0; itr < this.ncount; itr += 1) {
                if (this.inputOn[aitr][itr]) c += 1;
            }
            ntheta[aitr][2] = (float)c * this.ml[aitr][2] + 0.001f;
        }
     
        float[] emp = new float[3];
     
     
        for (itr = 0; itr < this.nsets; itr += 1) {
     
            emp[0] += ntheta[itr][0] / (ntheta[itr][0] + ntheta[itr][1] + ntheta[itr][2]) + 0.001f;
            if (Float.isNaN(emp[0])) emp[0] = 0.001f;
            if (Float.isInfinite(emp[0])) emp[0] = 0.001f;
         
            emp[1] += ntheta[itr][1] / (ntheta[itr][0] + ntheta[itr][1] + ntheta[itr][2]) + 0.001f;
            if (Float.isNaN(emp[1])) emp[1] = 0.001f;
            if (Float.isInfinite(emp[1])) emp[1] = 0.001f;
         
            emp[2] += ntheta[itr][2] / (ntheta[itr][0] + ntheta[itr][1] + ntheta[itr][2]) + 0.001f;
            if (Float.isNaN(emp[2])) emp[2] = 0.001f;
            if (Float.isInfinite(emp[2])) emp[2] = 0.001f;
        }
     
        emp[0] /= (float)this.nsets;
        emp[1] /= (float)this.nsets;
        emp[2] /= (float)this.nsets;
     
     
     
        this.theta[0] = (float)Math.sqrt((0.001f + this.theta[0]*emp[0])*2.0f);
        if (Float.isNaN(this.theta[0])) this.theta[0] = 0.001f;
        if (Float.isInfinite(this.theta[0])) this.theta[0] = 0.001f;
     
     
     
        this.theta[1] = (float)Math.sqrt((0.001f + this.theta[1]*emp[1])*2.0f);
        if (Float.isNaN(this.theta[1])) this.theta[1] = 0.001f;
        if (Float.isInfinite(this.theta[1])) this.theta[1] = 0.001f;
     
     
        this.theta[2] = (float)Math.sqrt((0.001f + this.theta[2]*emp[2])*2.0f);
        if (Float.isNaN(this.theta[2])) this.theta[2] = 0.001f;
        if (Float.isInfinite(this.theta[2])) this.theta[2] = 0.001f;
     
     
        return;
    }
 

    public void iterate() {
        int itr;
     
        for (itr = 0; itr < 27; itr += 1) {
            this.EStep();
            this.MStep();
        }
 
        return;
    }
 
    public boolean isOn() {
     
        boolean ret = (  (this.theta[2] >= CONSTANTS.THRESH && this.theta[0] <= 1.0f-CONSTANTS.THRESH) );
     
        return ret;
    }
 
    public float getOnValue() {
        return this.theta[2];
    }
 
}

//All iterations of EStep and MStep are O(N), as opposed to a NN's O(N^3)
//my MStep deviates from the actual algorithm

I've used the above code to smooth over varying variables such as the phase delta of a tone.

Can someone explain to me what is the intelligent part of neural nets?

In a neural net, the data has some memory to it, which if designed correctly may offer a machine the ability to make a decision based of of the numbers stored in the weights. In practice, however, I've used simple neural nets as a conditional. Say for instance, one would like to keep a variable true if it has been true in the last moment of time, regardless whether the input falls below a threshold. So, we could write the following code, as well:

public void setOn(boolean on) {
int itr;
this.onArray[0] = on;

for (itr = FRAME_RATE - 1; itr > 0; itr -= 1) {
this.onArray[itr] = this.onArray[itr - 1];
}

return;
}public boolean easyOn() {
int itr;

for (itr = 0; itr < FRAME_RATE; itr += 1) {
     if (this.onArray[itr]) return true;
}

return false;
}

Now if a condition exists that we now have an array of arrays of the above code structured as a neural network, would leaving the lights on waste electricity? Of course it would... regardless of my weird sense of humor, the reason for the derivation in back propagation is to pull down the network and allow negativity to exist.