How to Automate Data Extraction from a Scroll-Down Webpage?

  • Thread starter Thread starter vincent81
  • Start date Start date
  • Tags Tags
    Program Writing
Click For Summary
To create a program that extracts data from a scroll-down interface on a website, it is necessary to identify the HTML structure, particularly the combo bar tags that contain the URLs for each page. A web crawler or spider can be employed to navigate through these URLs systematically. The program should be designed to access each page, retrieve the data located at the same position on each page, and then compile this information into a text file. Existing tools or libraries for web scraping, such as Beautiful Soup or Scrapy in Python, may provide useful functionalities for this task.
vincent81
Messages
15
Reaction score
0
how do i go about writing this program?

This website has a scroll-down like the attached pic.
what i need to do is to go to each page in this scroll-down
and copy some data into a text file.
The data is located at the same position on the each page.

is there any such program found on the net?
 

Attachments

  • scrolldown.jpg
    scrolldown.jpg
    19.4 KB · Views: 598
Computer science news on Phys.org
Try looking for the combo bar html tags,
and use a spider to crawl the URL's found in the bar.
 
Thanks for the code...
:smile:
 
I’ve spent nearly my entire life online, and witnessed AI become integrated into our lives. It’s clear that AI is apart of us now whether we like it or not, unless your a anti tech cabin lover. AI has some form of control over your life. But what I’ve seen very recently is that people are loosing their ingenuity and deciding to use AI. I feel as if it’ll bleed into STEM which is kinda has already and, every idea or thought could become fully reliant on AI. Yeah AI makes life easier but at a...

Similar threads

Replies
21
Views
5K
  • · Replies 14 ·
Replies
14
Views
3K
  • · Replies 1 ·
Replies
1
Views
3K
  • · Replies 20 ·
Replies
20
Views
2K
Replies
7
Views
3K
  • · Replies 1 ·
Replies
1
Views
3K
  • · Replies 24 ·
Replies
24
Views
10K
  • · Replies 8 ·
Replies
8
Views
1K
Replies
10
Views
3K
  • · Replies 11 ·
Replies
11
Views
2K