Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Python Best way to get data from a website that is not obviously tabulated

  1. Feb 23, 2017 #1

    I'm trying to download and analyze the data from this link.

    I've used Python BS4 to read tabulated data before from a website, however this webpage is more complicated than any I have seen before. It's not set up as a table (at least that I can tell using inspect element). Is there a way to read each county and it's associated values (the ones that you mouse over to see)?
  2. jcsd
  3. Feb 23, 2017 #2
    Try this:
    1) Open that page in your web browser (I am using Firefox).
    2) Open this url on top of it: javascript:for(var i=0;i<dataset.length;i++){var ds=dataset;document.writeln("<br>"+i+":"+ds.paid+", "+ds.paidrank);}

    There were 3143 items (at 8:01pm EST).
    Fields in dataset include: paid, paidrank, sharerank, cfips, cname, homeval, and homevalrank. There may be others.
    You should have no problem modifying that javascript code to generate whatever table in whatever format you want.
  4. Feb 23, 2017 #3
    Hmm, it returns 3143 items, but they all say "undefined,undefined". And it seems to be stuck in an infinite loop.
  5. Feb 23, 2017 #4
    There's a typo:

    Oh, I see. The sequence [, i, ] was turned into an instruction to switch to italics.
    The statement ds=dataset should be ds=dataset with subscript i.
  6. Feb 23, 2017 #5
    Awesome! It's working now. Thank you!
  7. Feb 23, 2017 #6
    I modified the script to:
    Code (Javascript):
    for(var i=0; i < dataset.length; i++ ){
        var ds = dataset[i];
        document.writeln("<br>" + i + ": " + " cname: " + ds.cname + " paid: " + ds.paid + " paidrank: " + ds.paidrank + " cfips: " + ds.cfips + " homeval: " + ds.homeval + " homevalrank: " + ds.homevalrank + " share: " + ds.share + " sharerank: " + ds.sharerank);
    This is great, but the variable "cname" apparently only contains the county names, and does not include the state. This wouldn't be a problem, but sometimes two states can have a county with the same name, so when I go to categorize them into states later, I will have a problem with those counties with the same names.

    Do you know how I could get a list of all the properties of "dataset"? (looking for the one with the state name)

    I tried:
    Code (Javascript):
    for (var property in dataset) {
        if (dataset.hasOwnProperty(property)) {
    to no avail.
  8. Feb 23, 2017 #7
    When I tried, I got this:
    cname state cfips paid paidrank homeval homevalrank share sharerank

    The key is "var property in dataset[0]", not "var property in dataset".
  9. Feb 24, 2017 #8

    jack action

    User Avatar
    Science Advisor
    Gold Member

    Last edited: Feb 24, 2017
  10. Feb 24, 2017 #9
  11. Feb 24, 2017 #10
    I would use selenium to drive the web browser to parse it for me.
  12. Feb 26, 2017 #11
    In the past I've used Perl for this sort of thing. But even with Perl's nice regular expression handling, there was still the problem of web sites that would change some detail of the HTML, such as changing from <b> to <strong>, and then your search function had to be changed. Of course if it's a one-off, no worries about that. In any case, I used Perl, running from my desktop, along with SQLite, to take advantage of Perl's very nice database handling. It was all fairly easy, aside from the "moving target" issue.
  13. Feb 26, 2017 #12
    How do sites like Mint.com do stuff like this? There are a huge number of banks that they have to pull data from and it is all stored in a different way, and even if they changed one tag in the HTML it would cause the search to fail?
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted