More efficient way to do this using Python?

rollcast · Jun 18, 2012

Homework Statement

Its not really a homework problem as its just for a problem from an online Python tutorial but I thought it would fit in better here.

The problem is to find and then print out the all the web address links (clickable), from the source code of a web page.

The problem comes with a chunk of html code for you to find the links in it.

Html code:

Code:

<html xmlns="http://www.w3.org/1999/xhtml"><br/> <head><br/><title>Udacity</title> <br/></head><br/><br/><body> <br/><h1>Udacity</h1><br/><br/> <p><b>Udacity</b> is a private institution of <a href="[PLAIN]http://www.wikipedia.org/wiki/Higher_education">[/PLAIN] [/PLAIN]  higher education founded by</a> <a href="[PLAIN]http://www.wikipedia.org/wiki/Sebastian_Thrun">Sebastian[/PLAIN] [/PLAIN]  Thrun</a>, David Stavens, and Mike Sokolsky with the goal to provide university-level education that is "both high quality and low cost".<br/>It is the outgrowth of a free computer science class offered in 2011 through Stanford University. Currently, Udacity is working on its second course on building a search engine. Udacity was announced at the 2012 <a href="[PLAIN]http://www.wikipedia.org/wiki/Digital_Life_Design">Digital[/PLAIN] [/PLAIN]  Life Design</a> conference.</p><br/></body><br/></html>

The Attempt at a Solution

My code in python:

Code:

page = '<html xmlns="http://www.w3.org/1999/xhtml"><br/> <head><br/><title>Udacity</title> <br/></head><br/><br/><body> <br/><h1>Udacity</h1><br/><br/> <p><b>Udacity</b> is a private institution of <a href="[PLAIN]http://www.wikipedia.org/wiki/Higher_education">[/PLAIN] [/PLAIN]  higher education founded by</a> <a href="[PLAIN]http://www.wikipedia.org/wiki/Sebastian_Thrun">Sebastian[/PLAIN] [/PLAIN]  Thrun</a>, David Stavens, and Mike Sokolsky with the goal to provide university-level education that is "both high quality and low cost".<br/>It is the outgrowth of a free computer science class offered in 2011 through Stanford University. Currently, Udacity is working on its second course on building a search engine. Udacity was announced at the 2012 <a href="[PLAIN]http://www.wikipedia.org/wiki/Digital_Life_Design">Digital[/PLAIN] [/PLAIN]  Life Design</a> conference.</p><br/></body><br/></html>'
finished = False
start_link = 0
end_link = 0
previous_start_link = 0
while (True):
    start_link = page.find("<a href=", start_link + 1) + 9
    if (start_link < previous_start_link):
        break
    end_link = page.find('">', start_link)
    url = page[start_link:end_link]
    print url
    previous_start_link = start_link

However I realize that my implementation of a sort of "do-while" styled loop is quite chunky and probably could be refined along with my search method?

Thanks
Al

jedishrfu · Jun 18, 2012

the best way is to use an xml parser that will do the html parsing and present you with a tree of document components to look up what you want in this case <a> tags and what ther href attribute values are.

http://eli.thegreenplace.net/2012/03/15/processing-xml-in-python-with-elementtree/

rollcast · Jun 18, 2012

jedishrfu said:

the best way is to use an xml parser that will do the html parsing and present you with a tree of document components to look up what you want in this case <a> tags and what ther href attribute values are.

http://eli.thegreenplace.net/2012/03/15/processing-xml-in-python-with-elementtree/

Thanks for your reply.

The series eventually should teach you how to make a basic search engine in Python so that parser could be useful later on.

However, what could I do to the code I have made to make it more efficient or better than it is now without vastly changing how it works?

Thanks
Al

jedishrfu · Jun 18, 2012

well one thing I see is what if the "<a href=" is actually "<a href=" with extra spaces then your code will miss that url>

Can you search on say just http to find your strings? and then search for the next matching quote. Its more direct and avoids the "<a href=" extra spaces issue.

As far as your code goes it looks okay to me as a simple means of walking through the string of html code and finding the what you want.

An alternative to xml parsing would be a regular expression that extracted the string you wanted something along the lines of: /["]http[:].*["]/ to find all quoted strings that star with "http and end with "

import re

// get a list of strings matching the httpPattern
httplist = re.findall ( htpPattern, page )

However, parsing html in this manner is always tedious and error prone and that's why people tend to use xml parsing libraries and your teacher is probably trying the make that point with this exercise.

LudusRex · Jun 25, 2012

ison

I understand the importance of efficiency in problem-solving. In this case, using Python to find and print out web links from a webpage can definitely be done in a more efficient way.

One approach could be to use the BeautifulSoup library in Python, which is specifically designed for parsing HTML and XML files. It allows you to easily navigate through the HTML structure and extract the information you need, such as web links.

Another approach could be to use regular expressions to search for patterns in the HTML code that indicate a web link, and then extract the link using the re library in Python.

Both of these methods would likely be more efficient and concise than your current approach. I would recommend exploring and experimenting with different methods to find the most efficient solution for your problem.

More efficient way to do this using Python?

Homework Statement

The Attempt at a Solution

Related to More efficient way to do this using Python?

1. What is Python and why is it used for efficiency?

2. How can Python be used to improve efficiency in data analysis?

3. Is Python suitable for web development and how can it improve efficiency in this area?

4. Can Python be used for automating repetitive tasks and how does it save time?

5. Are there any tips for writing efficient code in Python?

Similar threads

Hot Threads

Recent Insights