Scraping a website for its words used

[bmorgan]

Is there a tool that you can enter the home URL of a website, then the software visits each page and extract the words it used?

 

[Mudundo07]

Yes most scrapers do that.. Look in github for free ones if you know python..

 

[Larry Igna]

Not exactly as you wrote it, it doesn't get all the pages if you provide URL to homepage, at least I don't know any script doing this. You would have to build it custom.
The closest thing for what you're looking for is Newspaper , a python module. It can build a set of articles from a news source (for example CNN), it extracts text, images or HTML. It has a lot of nice features like article summarization, keyword extraction and so on.

 

[bmorgan]

Yes most scrapers do that.. Look in github for free ones if you know python..

Thanks for the reply.

Although Python is new to me.
I can learn.

You mentioned most scrapers can do that, would scrapebox able to scrape a given list of URLs, and then extra the words, phrases it used in those URLs?

 

[bmorgan]

Not exactly as you wrote it, it doesn't get all the pages if you provide URL to homepage, at least I don't know any script doing this. You would have to build it custom.
The closest thing for what you're looking for is Newspaper , a python module. It can build a set of articles from a news source (for example CNN), it extracts text, images or HTML. It has a lot of nice features like article summarization, keyword extraction and so on.

Thanks for refering newspaper3k

I checked their demo here

It can fetch the whole content of a given news URL, like below.

What I am looking for is something that can not only download that article, but also can split the words and phases into a word list.

Is that something already there?

 

[Larry Igna]

Yes, you can use Nltk library, it has a tokenizer that can split paragraph into sentences or words. https://www.nltk.org/_modules/nltk/tokenize.html this is the tokenizer, more details on their website or just look on StackOverflow for some examples to get started.
Also, newspaper has a keyword extractor if that's what you're looking for. It can summarize and extract keywords using exactly the module mentioned above, Nltk.

 

[Kilroy_01]

Imacros will do that for you, and for free.