How to Automate Data Extraction from a Scroll-Down Webpage?

  • Thread starter Thread starter vincent81
  • Start date Start date
  • Tags Tags
    Program Writing
AI Thread Summary
To create a program that extracts data from a scroll-down interface on a website, it is necessary to identify the HTML structure, particularly the combo bar tags that contain the URLs for each page. A web crawler or spider can be employed to navigate through these URLs systematically. The program should be designed to access each page, retrieve the data located at the same position on each page, and then compile this information into a text file. Existing tools or libraries for web scraping, such as Beautiful Soup or Scrapy in Python, may provide useful functionalities for this task.
vincent81
Messages
15
Reaction score
0
how do i go about writing this program?

This website has a scroll-down like the attached pic.
what i need to do is to go to each page in this scroll-down
and copy some data into a text file.
The data is located at the same position on the each page.

is there any such program found on the net?
 

Attachments

  • scrolldown.jpg
    scrolldown.jpg
    19.4 KB · Views: 589
Computer science news on Phys.org
Try looking for the combo bar html tags,
and use a spider to crawl the URL's found in the bar.
 
Thanks for the code...
:smile:
 
Thread 'ChatGPT Examples, Good and Bad'
I've been experimenting with ChatGPT. Some results are good, some very very bad. I think examples can help expose the properties of this AI. Maybe you can post some of your favorite examples and tell us what they reveal about the properties of this AI. (I had problems with copy/paste of text and formatting, so I'm posting my examples as screen shots. That is a promising start. :smile: But then I provided values V=1, R1=1, R2=2, R3=3 and asked for the value of I. At first, it said...
Sorry if 'Profile Badge' is not the correct term. I have an MS 365 subscription and I've noticed on my Word documents the small circle with my initials in it is sometimes different in colour document to document (it's the circle at the top right of the doc, that, when you hover over it it tells you you're signed in; if you click on it you get a bit more info). Last night I had four docs with a red circle, one with blue. When I closed the blue and opened it again it was red. Today I have 3...
Back
Top