Scraping a website for its words used
- Thread starter bmorgan
- Start date
Larry Igna
Power Member
- Nov 25, 2016
- 663
- 717
Not exactly as you wrote it, it doesn't get all the pages if you provide URL to homepage, at least I don't know any script doing this. You would have to build it custom.
The closest thing for what you're looking for is Newspaper , a python module. It can build a set of articles from a news source (for example CNN), it extracts text, images or HTML. It has a lot of nice features like article summarization, keyword extraction and so on.
The closest thing for what you're looking for is Newspaper , a python module. It can build a set of articles from a news source (for example CNN), it extracts text, images or HTML. It has a lot of nice features like article summarization, keyword extraction and so on.
bmorgan
Junior Member
- Aug 25, 2011
- 173
- 52
Thanks for the reply.
Although Python is new to me.
I can learn.
You mentioned most scrapers can do that, would scrapebox able to scrape a given list of URLs, and then extra the words, phrases it used in those URLs?
bmorgan
Junior Member
- Aug 25, 2011
- 173
- 52
Thanks for refering newspaper3k
I checked their demo here
It can fetch the whole content of a given news URL, like below.
What I am looking for is something that can not only download that article, but also can split the words and phases into a word list.
Is that something already there?
Larry Igna
Power Member
- Nov 25, 2016
- 663
- 717
Yes, you can use Nltk library, it has a tokenizer that can split paragraph into sentences or words. https://www.nltk.org/_modules/nltk/tokenize.html this is the tokenizer, more details on their website or just look on StackOverflow for some examples to get started.
Also, newspaper has a keyword extractor if that's what you're looking for. It can summarize and extract keywords using exactly the module mentioned above, Nltk.
Also, newspaper has a keyword extractor if that's what you're looking for. It can summarize and extract keywords using exactly the module mentioned above, Nltk.
