Single layer neural network. What am I doing wrong?

1. Dec 27, 2011

GProgramer

I'm trying to implement this example http://www.cs.bham.ac.uk/~jxb/INC/l3.pdf (page 15)
I'm trying to do Iteration training, but it seems as if the results always converging to a steady error rate that is too large to be acceptable, the values centering around 0 while they should be close to -1 and +1.

I don't know if there's something wrong with the code, or I have the training concept misunderstood?
Code (Text):

close all;clc;
M=3;N=1;

X=[-1 1.0 0.1;-1 2.0 0.2; -1 0.1 0.3; -1 2.0 0.3; -1 0.2 0.4; -1 3.0 0.4; -1 0.1 0.5; -1 1.5 0.5; -1 0.5 0.6; -1 1.6 0.7];
X=X';
d=[-1;-1;1;-1;1;-1;1;-1;1;1];

Wp=rand([M,N]);
Wp=Wp'/sum(Wp(:));  % theta is 1 so sum of Wp and W needs to be <1
W=rand([M,N]);
W=W'/sum(W(:));
V1=zeros(1,10);  %Pre allocating for speed
Y1=zeros(1,10);
e=zeros(1,10);

while(1)

i=randi(length(X),1);
%---------------Feed forward---------------%
V1(i)=W*X(:,i);
Y1(i)=tanh(V1(i)/2);
e(i)=d(i)-Y1(i);

%------------Backward propagation---------%
delta1=e(i)*0.5*(1+Y1(i))*(1-Y1(i));

Wn(1,1)=W(1,1) + 0.1*(W(1,1)-Wp(1,1)) + 0.1*delta1*Y1(i);
Wn(1,2)=W(1,2) + 0.1*(W(1,2)-Wp(1,2)) + 0.1*delta1*Y1(i);
Wn(1,3)=W(1,3) + 0.1*(W(1,3)-Wp(1,3)) + 0.1*delta1*Y1(i);

Wp=W;
W=Wn;

figure(1);
stem(Y1);
axis([1 10 -1 1]);
drawnow;
end

2. Dec 27, 2011

D H

Staff Emeritus
First off, your backprop algorithm is just wrong. Do you have a reference for the algorithm you are using?

Even if you correct your update algorithm, a single layer perceptron is going to have a very, very hard time with this dataset. Make a scatter plot of this dataset. For example, make a graph with mass on the x axis, speed on the y axis. Mark each fighter with an F, bomber with a B. There are three clusters. Near the y axis there's a cluster of four light, fast fighters. Near the x axis there's a cluster of four slow, heavy bombers. The third cluster is going to be problematic for a single layer perceptron. The fighter with mass=1.6, speed=0.7 is very similar to the bomber with mass=1.5, speed=0.5.

Adding a two node hidden layer makes this problem much more amenable to a backprop neural network.

3. Dec 27, 2011