How can I log into a webpage using python and access data?

[dacruick]

Hello ladies and gentlemen,

I have grabbed a code for emailing using python. This isn't my end goal, but I think its a logical place to start and get used to the smtp library. here is the code

import smtplib

def prompt(prompt):
    return raw_input(prompt).strip()

fromaddr = prompt("From: ")
toaddrs  = prompt("To: ").split()
print "Enter message, end with ^D (Unix) or ^Z (Windows):"

# Add the From: and To: headers at the start!
msg = ("From: %s\r\nTo: %s\r\n\r\n"
       % (fromaddr, ", ".join(toaddrs)))
while 1:
    try:
        line = raw_input()
    except EOFError:
        break
    if not line:
        break
    msg = msg + line

print "Message length is " + repr(len(msg))

server = smtplib.SMTP('localhost')
server.set_debuglevel(1)
server.sendmail(fromaddr, toaddrs, msg)
server.quit()

I'm getting a server connection error called a socket error. on the line "server = smtplib.SMTP('localhost')"
It is telling me that my connection is being reset by a peer. Does anyone know what that means and what I can do to get rid of that error?

 

[Grep]

Do you have an SMTP server running on your PC? Most people don't.

If not, you'd maybe want to install one for testing purposes. Or you can find out what your normal SMTP server is and use that. Your ISP should have one. GMail also provides one.

Other than that, learning Python is on my to-do list, so I haven't looked closely at the code.

 

[dacruick]

How would I find my "normal" SMTP server?

 

[Grep]

How would I find my "normal" SMTP server?

That's what I'd call "opening a can of worms". It depends on your usual email account. There's so many possibilities it's impossible to say for sure without more information.

Do you use GMail? If so, then the server is probably smtp.gmail.com. Most servers are authenticated and require you to give them a username and password. There's such a call in smtplib:

SMTP.login(user, password)

You may also, possibly, need to use SSL to connect to gmail's smtp server. There's a call in smtplib for that as well.

If you use your ISP or generally go through a mail application on your PC, then you should be able to go in there and find the information on your account, which includes the smtp server. Every mail package is different, so where to find it, I can't say. It's quite likely to require a login as well. A server that doesn't require logging in is rare, which is good. Open mail servers are often abused by spammers, and admins that don't secure their mail servers are idiots. So you should expect that your mail server requires logging in.

If you use hotmail or something, I have no idea what the smtp server is. Like I said, there are so many possibilities I can't possibly enumerate them all.

 

[dacruick]

Hi Grep,

I am using gmail, and you were right about the smtp.gmail.com. Everything works fine. This didn't end up being as useful as I wanted it to be though. I wanted to use this as an exercise to get acquainted with accessing the internet with python. But I didn't use any lower level code to do this so its not really transferable to what I want to do.

I have data being logged on a server and I want to write a python program to grab it every hour and format it. I just don't know how to tell python to go to a certain site and log in kind of thing.

 

[Grep]

Hi Grep,

I am using gmail, and you were right about the smtp.gmail.com. Everything works fine. This didn't end up being as useful as I wanted it to be though. I wanted to use this as an exercise to get acquainted with accessing the internet with python. But I didn't use any lower level code to do this so its not really transferable to what I want to do.

I have data being logged on a server and I want to write a python program to grab it every hour and format it. I just don't know how to tell python to go to a certain site and log in kind of thing.

Hm, one solution would be an FTP server. If you put one up on the server (with proper security) and have the log copied into the ftp server directory, you could use python's FTP library to get it fairly simply. Without getting into details, that's one way.

Or you could have the server send an email containing the log to a specific account (perhaps putting up an SMTP server on the receiving end). Then you could have that server parse the log out of the mail messages.

Or perhaps you can use some kind of remote logging capability? On Unix type systems, syslogd (the system logging daemon) can do it, I think. Maybe Windows can as well.

There's always sharing the directory with the logs, and accessing it from another system.

Just tossing out ideas to consider. Might be of use. I'm not sure what resources you have available to you.

 

[dacruick]

yes thank you for all of your ideas. This is useful. I am going to see what I can do about this tomorrow, but is it alright if I repost on this thread in case I run into some problems?

 

[Grep]

yes thank you for all of your ideas. This is useful. I am going to see what I can do about this tomorrow, but is it alright if I repost on this thread in case I run into some problems?

Not a problem.

 

[dacruick]

Hi Grep,

I'm going to go in a different direction with this. I have to log into a webpage using python before I can access any of the data.

import urllib, urllib2, cookielib

username = 'username'
password = 'password'

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data = urllib.urlencode({'username' : username, 'password' : password, 'commit' : 'Log in' })
opener.open('site', login_data)
resp = opener.open('site I want to go to')

It seems that I can fill out the username and password fields with no problem, but the login button is giving me trouble.

The HTML for the login button is this

<input class="button" name="commit" onclick="alertForExplorerBrowserVersion(7, 3);" type="submit" value="Log in">

I figured that the 'commit' : 'Log in' tuple would activate the button but apparently not. Do you think you could shed some light?

EDIT: Hobolink.com is the site in case it is of any help