How to Automate Data Extraction from a Scroll-Down Webpage?

  • Thread starter Thread starter vincent81
  • Start date Start date
  • Tags Tags
    Program Writing
AI Thread Summary
To create a program that extracts data from a scroll-down interface on a website, it is necessary to identify the HTML structure, particularly the combo bar tags that contain the URLs for each page. A web crawler or spider can be employed to navigate through these URLs systematically. The program should be designed to access each page, retrieve the data located at the same position on each page, and then compile this information into a text file. Existing tools or libraries for web scraping, such as Beautiful Soup or Scrapy in Python, may provide useful functionalities for this task.
vincent81
Messages
15
Reaction score
0
how do i go about writing this program?

This website has a scroll-down like the attached pic.
what i need to do is to go to each page in this scroll-down
and copy some data into a text file.
The data is located at the same position on the each page.

is there any such program found on the net?
 

Attachments

  • scrolldown.jpg
    scrolldown.jpg
    19.4 KB · Views: 590
Computer science news on Phys.org
Try looking for the combo bar html tags,
and use a spider to crawl the URL's found in the bar.
 
Thanks for the code...
:smile:
 
I came across a video regarding the use of AI/ML to work through complex datasets to determine complicated protein structures. It is a promising and beneficial use of AI/ML. AlphaFold - The Most Useful Thing AI Has Ever Done https://www.ebi.ac.uk/training/online/courses/alphafold/an-introductory-guide-to-its-strengths-and-limitations/what-is-alphafold/ https://en.wikipedia.org/wiki/AlphaFold https://deepmind.google/about/ Edit/update: The AlphaFold article in Nature John Jumper...
Thread 'Urgent: Physically repair - or bypass - power button on Asus laptop'
Asus Vivobook S14 flip. The power button is wrecked. Unable to turn it on AT ALL. We can get into how and why it got wrecked later, but suffice to say a kitchen knife was involved: These buttons do want to NOT come off, not like other lappies, where they can snap in and out. And they sure don't go back on. So, in the absence of a longer-term solution that might involve a replacement, is there any way I can activate the power button, like with a paperclip or wire or something? It looks...
Back
Top