Is There a Tool to Efficiently Download Articles from a Newspaper's Website?

  • Thread starter Thread starter mech-eng
  • Start date Start date
  • Tags Tags
    Articles
Click For Summary

Discussion Overview

The discussion revolves around the challenges and methods for efficiently downloading articles from a newspaper's website, particularly focusing on the works of specific journalists. Participants explore various tools and considerations related to copyright and access limitations.

Discussion Character

  • Exploratory
  • Technical explanation
  • Debate/contested

Main Points Raised

  • One participant expresses a desire to download numerous articles from a specific author’s newspaper website and inquires about practical tools for this purpose.
  • Another participant suggests command line tools like curl and wget for Linux as potential solutions for downloading webpages and their references, while also cautioning against capturing copyrighted material.
  • A follow-up post reiterates the suggestion of curl and wget, questioning the implications of accessing copyrighted material if the intention is merely for personal use and archiving.
  • Some participants emphasize the importance of personal judgment regarding the legality of downloading articles, particularly if there is a plan to republish them.
  • One participant notes that many newspapers allow non-subscribers to access a limited number of articles and suggests printing or saving PDFs of articles as a viable alternative for later access.

Areas of Agreement / Disagreement

Participants express differing views on the legality and ethics of downloading articles, with some advocating for personal discretion while others highlight potential copyright issues. The discussion does not reach a consensus on the best approach to downloading articles.

Contextual Notes

Participants mention the potential for articles to be removed if a website shuts down or if an author leaves a newspaper, indicating a concern for preserving access to content over time. There are also references to the limitations of access based on subscription models and copyright restrictions.

mech-eng
Messages
826
Reaction score
13
I would like to download articles of my favorite journalists. But they are in their newspaper's website. There are hundrends of article for one author. Is there a practical way to download a author's articles. I have to click hundreds of times. Is there any program for this case? For years ago I heard a type of programs called vampire programs. They could download the website so that programs' users can surf on that websites offline. Do you have any idea?

Thank you.
 
Computer science news on Phys.org
jedishrfu said:
There are command line tools such as curl and wget for linux that can pull down webpages and what they reference.

https://en.wikipedia.org/wiki/CURL

https://en.wikipedia.org/wiki/Wget

Remember though we don't want to capture copyrighted material.

If you just read them in the future is this related to copyrighted material? Those websites might be shutdown in the future so having an archive would be better. I use windows can I still use them?

Thank you.
 
That's for you to answer not me. If you feel its wrong then don't do it.

If you plan to post or republish them then that may go over the line.
 
jedishrfu said:
That's for you to answer not me. If you feel its wrong then don't do it.

If you plan to post or republish them then that may go over the line.

No, I just want them not to be terminated in the future and of course reading them. If the websites shutdown or author leaves from newspaper the articles will be deleted i.e terminated.

Thank you.
 
Most newspapers I know of allow non-subscribers to access some number of articles/ some time period.
Once you access an article you could print or make a pdf of it for later.

If the author were to leave a newspaper, they would not throw away their old articles.
 
  • Like
Likes   Reactions: davenn

Similar threads

  • · Replies 2 ·
Replies
2
Views
2K
  • · Replies 3 ·
Replies
3
Views
7K
  • · Replies 1 ·
Replies
1
Views
2K
Replies
4
Views
2K
  • · Replies 14 ·
Replies
14
Views
4K
  • · Replies 17 ·
Replies
17
Views
16K
  • · Replies 14 ·
Replies
14
Views
4K
  • · Replies 19 ·
Replies
19
Views
4K
  • · Replies 6 ·
Replies
6
Views
4K
Replies
2
Views
3K