Logistic regression: Stochastic Gradient Ascent (in python)

[NATURE.M]

So I've been following through a online course in machine learning offered by Stanford university. I have been recently reading up on logistic regression and stochastic gradient ascent. Here is a link to the original notes: http://cs229.stanford.edu/notes/cs229-notes1.pdf (pages 16-19).

Here is my code. The issue I'm having is that the log likelihood function seems to be constantly decreasing (becoming more negative) rather than increasing after every change in theta. The value for alpha (learning rate) I'm using is 0.0001 which is small, however anything larger will produce nan values as output from the log likelihood function. I spent quite some time playing around with this value and looking over my code and I can't seem to figure out what's going wrong.

import numpy as np

def logregg(X, y, alpha, iterations):
    #output parameter vector

    (m, n) = np.shape(X)
    theta = np.array([0] * n, 'float')
    lin = np.dot(X, theta)
    hyp = 1./(1 + np.exp(-1*lin))
    i = 0
 
    #stochastic gradient ascent
    while i < iterations:
        for j in range(m):
            theta = theta + alpha*(y[j] - hyp[j])*X[j,:]
            print loglik(X, y, theta)
        i+=1
        
    return thetadef loglik(X, y, theta):
    lin = np.dot(X, theta)
    hyp = 1./(1 + np.exp(-1 * lin))
    first = y*np.log(hyp)
    second = (1-y)*np.log(1-hyp)
 
    return np.sum(first + second)

 

[wilsont23274]

X = np.array([[1,2], [2,3], [3,4], [4,5], [5,6]])y = np.array([1, 0, 1, 0, 1])theta = logregg(X, y, 0.0001, 10)print theta