Automatic Market Research Program

Click For Summary
SUMMARY

The discussion centers on a project involving web scraping to gather product information from hktvmall.com. The goal is to extract similar products, including their prices and descriptions, from a provided Excel list. Participants shared insights on tools and methods for effective web scraping, emphasizing the importance of using libraries like Beautiful Soup and Scrapy for Python to automate the data collection process.

PREREQUISITES
  • Familiarity with Python programming
  • Understanding of web scraping techniques
  • Knowledge of HTML structure for data extraction
  • Experience with libraries such as Beautiful Soup and Scrapy
NEXT STEPS
  • Research how to use Beautiful Soup for parsing HTML
  • Learn about Scrapy framework for large-scale web scraping
  • Explore data storage options for scraped data, such as CSV or databases
  • Investigate ethical considerations and legal implications of web scraping
USEFUL FOR

This discussion is beneficial for data analysts, web developers, and students involved in market research who need to automate the collection of product data from e-commerce websites.

Leo_Chau_430
Messages
8
Reaction score
1
Homework Statement
I was given an Excel that contains a list of products, my teacher would like us to scrap through the website hktvmall.com to find respective similar products and list out their prices and description.

FYI: https://docs.google.com/spreadsheets/d/1nCPHcuE20tf-Al_Q7wi0X_cqzQrV6VKh-eofGfVcER8/edit?usp=sharing
Relevant Equations
N/A
I was given an Excel that contains a list of products, my teacher would like us to scrap through the website hktvmall.com to find respective similar products and list out their prices and description.

FYI:
 
Last edited by a moderator:
Physics news on Phys.org

Similar threads

Cournot Model in a Duopoly Market
  • · Replies 3 ·
Replies
3
Views
1K
How do you deal with your Amazon wishlist?
  • · Replies 1 ·
Replies
1
Views
2K
Programs Take a 5th year to do more research?
  • · Replies 3 ·
Replies
3
Views
2K
Programs Difficult Decision: MS Operations Research or Software Development?
  • · Replies 3 ·
Replies
3
Views
3K
Comp Sci PSPice Lab Exercise - Differentiator (Circuit)
  • · Replies 4 ·
Replies
4
Views
2K
Admissions What are my chances of getting into a competitive physics PhD program?
  • · Replies 7 ·
Replies
7
Views
3K
Schools How to Navigate Applying to HEPTH PhD Programs in Europe vs. the US?
  • · Replies 12 ·
Replies
12
Views
1K
Admissions Should I submit an 840 PGRE in my applications to top US PhD programs?
  • · Replies 2 ·
Replies
2
Views
2K
Programs Questions Regarding Undergrad REU Programs
  • · Replies 4 ·
Replies
4
Views
2K
Comp Sci Problem with C++ Information Structures, Pointers, Arrays
  • · Replies 3 ·
Replies
3
Views
2K