Automated data collection from the net database

Click For Summary
SUMMARY

The discussion focuses on automating data collection from the Singapore Pools website using a specific code snippet. The provided code is designed to extract numerical data from a webpage and save it into a text file located at "C:\Textfile.txt". The code utilizes a single-dimensional array to store characters and employs conditional checks to ensure only valid numerical data is printed to the output file. This method is applicable to similar web scraping tasks involving structured data extraction.

PREREQUISITES
  • Basic understanding of web scraping techniques
  • Familiarity with programming concepts, particularly arrays
  • Knowledge of file handling in programming languages
  • Experience with numerical data validation methods
NEXT STEPS
  • Research web scraping tools like Beautiful Soup for Python
  • Learn about regular expressions for data validation
  • Explore file I/O operations in programming languages such as Python or Java
  • Investigate automated data collection frameworks like Scrapy
USEFUL FOR

Web developers, data analysts, and anyone interested in automating data extraction from websites will benefit from this discussion.

vincent81
Messages
15
Reaction score
0
hi ..i have a little problem here...if u go to this site,
http://www.singaporepools.com.sg/fd_results3yrs/
u see the scroll down button...there are a lot of dates...each date links to a page in which i want to copy some data into a text file.
and this apples to all other dates...

anyone know any program which can do this job?
 
Computer science news on Phys.org
load into array(single dimensional, one cell per character)
open "C:\Textfile.txt" for output as #1
for x = 2 to ubound(array()) - 2
if isnumerical(array(x)) then
if isnumerical(array(x + 1)) then
if isnumerical(array(x + 2)) then
if isnumerical(array(x - 1)) then
if isnumerical(array(x - 2)) then
print #1, array(x)
end if
end if
end if
end if
end if
next x
close #1

Hopefully this code will help you
 

Similar threads

  • · Replies 1 ·
Replies
1
Views
2K
  • · Replies 11 ·
Replies
11
Views
2K
Replies
65
Views
5K
Replies
1
Views
1K
  • · Replies 8 ·
Replies
8
Views
3K
  • · Replies 5 ·
Replies
5
Views
2K
  • · Replies 3 ·
Replies
3
Views
7K
  • · Replies 5 ·
Replies
5
Views
3K
  • · Replies 2 ·
Replies
2
Views
4K
  • · Replies 1 ·
Replies
1
Views
3K