Advise on writing program to collect visual sighting reports online autonomously

[taylaron]

Greetings, I'm an undergraduate electrical engineer with minimal programming experience and I'm here because I would like some advice as to how I should approach writing a program I'm interested in. I'm specifically interested in what programming language I should learn and use.

The purpose of the program is to autonomously search the internet (with the help of Google's search engine) for user reports of a specific visual phenomenon called ball lightning. This program would be for the purpose of data collection in respect to the date and location (eventually latitude and longitude) of the sightings. This program would compile a matrix of report data of the previously specified 2 data points.

An example of the data I expect the program to analyze is the following report from http://amasci.com/weird/unusual/bl.html"

"On tuesday 29th june 2010 at 12.10am aprox, my husband and myself think we both witnessed 'ball lightning'. it was a warm night with no wind and we were sat in the back garden, my husband was first to see it and told me to look to my left. we watched a ball,(the size of a grapefruit), of brilliant blue white light travel in a horizontal direction above the roofs of our neighbours house and then our house. It made no sound then just disapered! I must admit it did scare me as I've never seen anything like it. We ruled out a meteor as it had no tail nor sound and it traveled in a horizontal straight line, i havnt seen a meteor do this!
amanda bromley
USA - Wednesday, June 30, 2010 at 16:20:39 (PDT)"

I would expect the program to be able to single out this report from the whole page of them, look for key words including "On Tuesday 29th June 2010 at 12.10am" and "USA". Then input the date and location into a matrix.

Another, more challenging example:

Back about 1948 myself and my family were sitting around the table eating. It was during a thunder storm.We heard a loud thunder clap and the telephone rang once. An orange gas-like ball about the size of a grape fruit came out of the phone reciever and sailed right across the table between us went right to the kitchen sink on the other side of the room.This happened another one other time also. I later found that the telephone ground to a water pipe was not connected.
charles w. ward <none>
hopkinton, ma USA - Thursday, February 03, 2011 at 17:44:24 (PST)

The program would need to recognize that there was no specific location listed in the sighting, but the keywords "family, table, kitchen" would indicate the sighting was at the reporter's home and would resort to the location listed next to the date the sighting was listed on the website; opposed to another location listed in the report.
Data collected from this report:
Date: 1948
location: Hopkinton, MA, USA


What programming language would you recommend I use to write such a program and what is the level of difficulty of such an endeavor?

Thank you for your time,
-Taylaron

 

[Future Bruno]

The language that I would recommend for this project is Python. It is a versatile programming language that is relatively easy to learn and use. Python is also well suited for web scraping and automated data collection, which are ideal for your project. You can also find many tutorials online to help you get started with Python. The difficulty of this project really depends on the complexity of the data you want to collect. If it is straightforward, then it should be relatively easy to write a program to do the job. However, if you need to take into account more complex scenarios, such as the second example you gave, then it might be a bit more challenging.