Tool to mass download foldered data

[dextercioby]

Hi, I need to know if there's any way (AI or not) in which I can quickly download data that is (freely, no copyright issue) available on a website in form of a list of 160 links, each leading to list of zipped files. Basically, I need the full list of zipped files. Is there a way, other than download them from each 160 links?

 

[jedishrfu]

Maybe the wget command can do it.

wget --recursive --level=1 \
  --page-requisites \
  --convert-links \
  --adjust-extension \
  --no-parent \
  --domains example.com \
  https://example.com/page

 

[dextercioby]

I can try, but how and where do I type it?

 

[jedishrfu]

It's a command so open a command session and type it there.

You can test if its available via the which command:

~ x which wget

and it will either say which: no wget in (list of directories from the the path environment parameter) or it will show where its located.

Look for examples online or ask your AI to tell you how to use it.

 

[pbuk]

(freely, no copyright issue) available on a website

This is called web crawling and a few caveats apply:

• It may be prohibited by the website's terms and conditions.
• If not permitted by the website owner it may be a criminal offence.
• The website owner may take steps to prevent this, often by using a third party service provider (have you ever seen a "we have detected unusual traffic" screen and had to pass a Captcha test?)
• @jedishrfu's solution requires access to a Linux command prompt and is particularly likely to trigger any protection methods used by the website. It could even result in you being unable to access other websites that use the same service provider for protecting it in this way.

Have you tried asking the website owner if they have a single archive of all their files you can download?

Good luck!

 

[dextercioby]

Thanks to both of you. The issue is now solved. Someone used some smart AI script not to download, but to map the zipped content of thousands and thousands of .csv-s on another website that I can access and query just like a regular database.

 

[WWGD]

I can try, but how and where do I type it?

Powershell .

 

[WWGD]

This is called web crawling and a few caveats apply:

• It may be prohibited by the website's terms and conditions.
• If not permitted by the website owner it may be a criminal offence.
• The website owner may take steps to prevent this, often by using a third party service provider (have you ever seen a "we have detected unusual traffic" screen and had to pass a Captcha test?)
• @jedishrfu's solution requires access to a Linux command prompt and is particularly likely to trigger any protection methods used by the website. It could even result in you being unable to access other websites that use the same service provider for protecting it in this way.

Have you tried asking the website owner if they have a single archive of all their files you can download?

Good luck!

Or permission from the admin to implement the method suggested by @jedishrfu and @pbuk

 

[jedishrfu]

Sadly, I don't have that power and wouldn't want it.

 

[WWGD]

BTW, you may have similar issues with scraping, if you don't limit your rate or fail to ask admin before.

 

[A.T.]

BTW, you may have similar issues with scraping, if you don't limit your rate or fail to ask admin before.

A randomized delay between requests can help.

 

[A.T.]

Maybe the wget command can do it.

wget --recursive --level=1 \
  --page-requisites \
  --convert-links \
  --adjust-extension \
  --no-parent \
  --domains example.com \
  https://example.com/page

For static websites I often used a repeated combination of wget (to download the html and finaly data) & grep (to extract the links from html), to get through multiple layers of link pages to the download links.

For dynamically generated link pages you need a tool/library that emulates a web browser and executes all client side scripts (without showing any graphics).