Single layer neural network. What am I doing wrong?

[GProgramer]

I'm trying to implement this example http://www.cs.bham.ac.uk/~jxb/INC/l3.pdf (page 15)
I'm trying to do Iteration training, but it seems as if the results always converging to a steady error rate that is too large to be acceptable, the values centering around 0 while they should be close to -1 and +1.

I don't know if there's something wrong with the code, or I have the training concept misunderstood?

	close all;clc;
	M=3;N=1;

	X=[-1 1.0 0.1;-1 2.0 0.2; -1 0.1 0.3; -1 2.0 0.3; -1 0.2 0.4; -1 3.0 0.4; -1 0.1 0.5; -1 1.5 0.5; -1 0.5 0.6; -1 1.6 0.7];
	X=X';
	d=[-1;-1;1;-1;1;-1;1;-1;1;1];

	Wp=rand([M,N]);
	Wp=Wp'/sum(Wp(:));  % theta is 1 so sum of Wp and W needs to be <1
	W=rand([M,N]);
	W=W'/sum(W(:));
	V1=zeros(1,10);  %Pre allocating for speed
	Y1=zeros(1,10);
	e=zeros(1,10);

	while(1)
		
	i=randi(length(X),1);
	%---------------Feed forward---------------%    
	V1(i)=W*X(:,i);
	Y1(i)=tanh(V1(i)/2);
	e(i)=d(i)-Y1(i);


	%------------Backward propagation---------%
	delta1=e(i)*0.5*(1+Y1(i))*(1-Y1(i));

	Wn(1,1)=W(1,1) + 0.1*(W(1,1)-Wp(1,1)) + 0.1*delta1*Y1(i);
	Wn(1,2)=W(1,2) + 0.1*(W(1,2)-Wp(1,2)) + 0.1*delta1*Y1(i);
	Wn(1,3)=W(1,3) + 0.1*(W(1,3)-Wp(1,3)) + 0.1*delta1*Y1(i);

	Wp=W;
	W=Wn;

	figure(1);
	stem(Y1);
	axis([1 10 -1 1]);
	drawnow;
	end

 

[D H]

First off, your backprop algorithm is just wrong. Do you have a reference for the algorithm you are using?

Even if you correct your update algorithm, a single layer perceptron is going to have a very, very hard time with this dataset. Make a scatter plot of this dataset. For example, make a graph with mass on the x axis, speed on the y axis. Mark each fighter with an F, bomber with a B. There are three clusters. Near the y-axis there's a cluster of four light, fast fighters. Near the x-axis there's a cluster of four slow, heavy bombers. The third cluster is going to be problematic for a single layer perceptron. The fighter with mass=1.6, speed=0.7 is very similar to the bomber with mass=1.5, speed=0.5.

Adding a two node hidden layer makes this problem much more amenable to a backprop neural network.

 

[GProgramer]

First off, your backprop algorithm is just wrong. Do you have a reference for the algorithm you are using?

Even if you correct your update algorithm, a single layer perceptron is going to have a very, very hard time with this dataset. Make a scatter plot of this dataset. For example, make a graph with mass on the x axis, speed on the y axis. Mark each fighter with an F, bomber with a B. There are three clusters. Near the y-axis there's a cluster of four light, fast fighters. Near the x-axis there's a cluster of four slow, heavy bombers. The third cluster is going to be problematic for a single layer perceptron. The fighter with mass=1.6, speed=0.7 is very similar to the bomber with mass=1.5, speed=0.5.

Adding a two node hidden layer makes this problem much more amenable to a backprop neural network.

Thank you for your detailed reply. I realized my mistake was taking the Y1 instead of X when readjusting the weights.

Currently it's more or less working (with a couple of the training values having around 30% error rate)

This particular set isn't what I'm aiming for, so it's why I didn't bother doing a detailed graph for it.

I realize that a single layer perceptron isn't enough, and I was just making one so as to test the algo if it's working. I am currently making a 2 hidden layer network, with a dynamic number of neurons in the hidden layers, and I will test it on a sampled cosine function, drawing the output function and the desired function over the same graph so as to compare.

May I ask exactly what is wrong with the algo? If you are talking about the hidden layer deltas, I've yet to add them.

Thank you again for the great reply!