- #1
- 7,330
- 11,193
- TL;DR Summary
- Code outputs wanted columns but not the data
Hi, trying to scrape a page :
https://www.hofstede-insights.com/wp-json/v1/countryI get the list of columns I want, but not the data assigned to the columns in the page that is being scraped.
Code compiles with no problem. Output is:
adjective id idv ind ivr lto mas name pdi slug title uai
______________________________________________________________________________________
And no error message. But no data.
I used the same code with no problem for another page. Where did I go wrong here?
https://www.hofstede-insights.com/wp-json/v1/countryI get the list of columns I want, but not the data assigned to the columns in the page that is being scraped.
Python:
from bs4 import BeautifulSoup
import pandas as pd
import requests
url = "https://www.hofstede-insights.com/wp-json/v1/country"page = requests.get(url)
soup = BeautifulSoup(page.text)
adjective= []
name = []
id= []
idv= []
ind = []
ivr = []
lto = []
mas = []
pdi = []
slug= []
title= []
uai= []for row in soup.find_all('tr')[1:]:
col = row.find_all('td')
column_1 = col[1].string.strip()
adjective.append(column_1)
column_2 = col[2].string.strip()
name.append(column_2)
column_3 = col[3].string.strip()
id.append(column_3)
column_4 = col[4].string.strip()
idv.append(column_4)
column_5 = col[5].string.strip()
ind.append(column_5)
column_6 = col[6].string.strip()
ivr.append(column_6)
column_7 = col[7].string.strip()
ito.append(column_7)
column_ =col[8].string.strip()
mas.append(column_8)
column_ =col[9 ].string.strip()
pdi.append(column_9)
column_ =col[10 ].string.strip()
slug.append(column_10)
column_ =col[11].string.strip()
title.append(column_11)
column_ =col[12].string.strip()
uai.append(column_12)
columns = {
"adjective":adjective,
"name" :name,
"id": id,
"idv": idv,
"ind" :ind,
"ivr" : ivr,
"lto" : lto,
"mas" : mas,
"pdi" : pdi,
"slug":slug,
"title":title,
"uai":uai
}
df = pd.DataFrame(columns)
df.to_csv("somefile.csv",index = False)
adjective id idv ind ivr lto mas name pdi slug title uai
______________________________________________________________________________________
And no error message. But no data.
I used the same code with no problem for another page. Where did I go wrong here?
Last edited: