More efficient way to do this using Python?

  • Context: Python 
  • Thread starter Thread starter rollcast
  • Start date Start date
  • Tags Tags
    Python
Click For Summary

Discussion Overview

The discussion revolves around improving a Python code snippet designed to extract web address links from HTML source code. Participants explore various methods for parsing HTML, including string manipulation and the use of XML parsers.

Discussion Character

  • Technical explanation
  • Debate/contested
  • Mathematical reasoning

Main Points Raised

Areas of Agreement / Disagreement

There is no consensus on the best approach to improve the code. Multiple competing views on methods for parsing HTML remain, including string manipulation, regular expressions, and XML parsing.

Contextual Notes

Participants note limitations in the original code, such as its sensitivity to formatting variations in HTML and the potential for error when using string methods. The discussion does not resolve these issues.
rollcast
Messages
403
Reaction score
0

Homework Statement


Its not really a homework problem as its just for a problem from an online Python tutorial but I thought it would fit in better here.

The problem is to find and then print out the all the web address links (clickable), from the source code of a web page.

The problem comes with a chunk of html code for you to find the links in it.

Html code:
Code: 
<html xmlns="http://www.w3.org/1999/xhtml"><br/> <head><br/><title>Udacity</title> <br/></head><br/><br/><body> <br/><h1>Udacity</h1><br/><br/> <p><b>Udacity</b> is a private institution of <a href="[PLAIN]http://www.wikipedia.org/wiki/Higher_education">[/PLAIN] [/PLAIN]  higher education founded by</a> <a href="[PLAIN]http://www.wikipedia.org/wiki/Sebastian_Thrun">Sebastian[/PLAIN] [/PLAIN]  Thrun</a>, David Stavens, and Mike Sokolsky with the goal to provide university-level education that is "both high quality and low cost".<br/>It is the outgrowth of a free computer science class offered in 2011 through Stanford University. Currently, Udacity is working on its second course on building a search engine. Udacity was announced at the 2012 <a href="[PLAIN]http://www.wikipedia.org/wiki/Digital_Life_Design">Digital[/PLAIN] [/PLAIN]  Life Design</a> conference.</p><br/></body><br/></html>

The Attempt at a Solution



My code in python:

Code: 
page = '<html xmlns="http://www.w3.org/1999/xhtml"><br/> <head><br/><title>Udacity</title> <br/></head><br/><br/><body> <br/><h1>Udacity</h1><br/><br/> <p><b>Udacity</b> is a private institution of <a href="[PLAIN]http://www.wikipedia.org/wiki/Higher_education">[/PLAIN] [/PLAIN]  higher education founded by</a> <a href="[PLAIN]http://www.wikipedia.org/wiki/Sebastian_Thrun">Sebastian[/PLAIN] [/PLAIN]  Thrun</a>, David Stavens, and Mike Sokolsky with the goal to provide university-level education that is "both high quality and low cost".<br/>It is the outgrowth of a free computer science class offered in 2011 through Stanford University. Currently, Udacity is working on its second course on building a search engine. Udacity was announced at the 2012 <a href="[PLAIN]http://www.wikipedia.org/wiki/Digital_Life_Design">Digital[/PLAIN] [/PLAIN]  Life Design</a> conference.</p><br/></body><br/></html>'
finished = False
start_link = 0
end_link = 0
previous_start_link = 0
while (True):
    start_link = page.find("<a href=", start_link + 1) + 9
    if (start_link < previous_start_link):
        break
    end_link = page.find('">', start_link)
    url = page[start_link:end_link]
    print url
    previous_start_link = start_link

However I realize that my implementation of a sort of "do-while" styled loop is quite chunky and probably could be refined along with my search method?

Thanks
Al
 
Last edited by a moderator:
Technology news on Phys.org
jedishrfu said:
the best way is to use an xml parser that will do the html parsing and present you with a tree of document components to look up what you want in this case <a> tags and what ther href attribute values are.

http://eli.thegreenplace.net/2012/03/15/processing-xml-in-python-with-elementtree/

Thanks for your reply.

The series eventually should teach you how to make a basic search engine in Python so that parser could be useful later on.

However, what could I do to the code I have made to make it more efficient or better than it is now without vastly changing how it works?

Thanks
Al
 
well one thing I see is what if the "<a href=" is actually "<a href=" with extra spaces then your code will miss that url>

Can you search on say just http to find your strings? and then search for the next matching quote. Its more direct and avoids the "<a href=" extra spaces issue.

As far as your code goes it looks okay to me as a simple means of walking through the string of html code and finding the what you want.

An alternative to xml parsing would be a regular expression that extracted the string you wanted something along the lines of: /["]http[:].*["]/ to find all quoted strings that star with "http and end with "

import re

// get a list of strings matching the httpPattern
httplist = re.findall ( htpPattern, page )

However, parsing html in this manner is always tedious and error prone and that's why people tend to use xml parsing libraries and your teacher is probably trying the make that point with this exercise.
 

Similar threads

Python Does anyone know about python flask and wtf forms?....
  • · Replies 2 ·
Replies
2
Views
2K
HTML/CSS Python - HTML : Isn't this a dictionary ?
  • · Replies 2 ·
Replies
2
Views
2K
Why Does My Python URL Extraction Code Not Work?
  • · Replies 3 ·
Replies
3
Views
3K
HTML/CSS Background-position CSS confusion with percentage values?
  • · Replies 9 ·
Replies
9
Views
3K
HTML/CSS HTML and PHP issue: display HTML form variables using PHP echo
  • · Replies 6 ·
Replies
6
Views
7K
How to Create a PHP Random Number Generator for a Coin Toss?
  • · Replies 1 ·
Replies
1
Views
8K
Mathematica How to type mathematical equations using LaTeX
  • · Replies 3 ·
Replies
3
Views
2K
HTML/CSS Cascading Style Sheets in XHTML 1.0 STRICT
  • · Replies 4 ·
Replies
4
Views
3K
Undergrad PHP Random number generator (Need help)
  • · Replies 3 ·
Replies
3
Views
9K
PHP Using PHP and cURL to submit POST requests to a website
  • · Replies 7 ·
Replies
7
Views
6K