How to Automate Data Extraction from a Scroll-Down Webpage?

  • Thread starter Thread starter vincent81
  • Start date Start date
  • Tags Tags
    Program Writing
AI Thread Summary
To create a program that extracts data from a scroll-down interface on a website, it is necessary to identify the HTML structure, particularly the combo bar tags that contain the URLs for each page. A web crawler or spider can be employed to navigate through these URLs systematically. The program should be designed to access each page, retrieve the data located at the same position on each page, and then compile this information into a text file. Existing tools or libraries for web scraping, such as Beautiful Soup or Scrapy in Python, may provide useful functionalities for this task.
vincent81
Messages
15
Reaction score
0
how do i go about writing this program?

This website has a scroll-down like the attached pic.
what i need to do is to go to each page in this scroll-down
and copy some data into a text file.
The data is located at the same position on the each page.

is there any such program found on the net?
 

Attachments

  • scrolldown.jpg
    scrolldown.jpg
    19.4 KB · Views: 590
Computer science news on Phys.org
Try looking for the combo bar html tags,
and use a spider to crawl the URL's found in the bar.
 
Thanks for the code...
:smile:
 
This week, I saw a documentary done by the French called Les sacrifiés de l'IA, which was presented by a Canadian show Enquête. If you understand French I recommend it. Very eye-opening. I found a similar documentary in English called The Human Cost of AI: Data workers in the Global South. There is also an interview with Milagros Miceli (appearing in both documentaries) on Youtube: I also found a powerpoint presentation by the economist Uma Rani (appearing in the French documentary), AI...
Back
Top