ScrapeBox: how do I download/scrape a whole website?

phr0zt said:

@loopline knows more about SB than anyone (except maybe the developer), he may have some insight for you

Click to expand...

loopline said:

Thanks mate.


OP yes its doable if the pages do Not require you to be logged in to see them AND the info is viewable when javascript is turned off.

However you are talking about first extracting all the pages of the site, which isn't too hard.

Then after that you need to build a custom data scraper module. Presumably the data is formatted the same across all the pages, so it should work.

However each piece of data is likley to require its own mask. Thats fine and all, but each mask will output its data on its own line

So like

name
phone
email
website
etc...

Also any time a piece of data is not found its skipped, so you may have missing lines etc.. You can link the url the data is found with to each piece of data, so that can make distinguishing it easier. But you should test it on a page or two to make sure your happy with what you get.

If you are wanting it in a neat excel sheet when you are done, you would need to do post processing to get the data out of the 1 piece per line in a basic text file into an excel sheet. But sure, as long as the above criteria are met, you can do it.

Two videos

Click to expand...

loopline said:

Thanks mate.


OP yes its doable if the pages do Not require you to be logged in to see them AND the info is viewable when javascript is turned off.

However you are talking about first extracting all the pages of the site, which isn't too hard.

Then after that you need to build a custom data scraper module. Presumably the data is formatted the same across all the pages, so it should work.

However each piece of data is likley to require its own mask. Thats fine and all, but each mask will output its data on its own line

So like

name
phone
email
website
etc...

Also any time a piece of data is not found its skipped, so you may have missing lines etc.. You can link the url the data is found with to each piece of data, so that can make distinguishing it easier. But you should test it on a page or two to make sure your happy with what you get.

If you are wanting it in a neat excel sheet when you are done, you would need to do post processing to get the data out of the 1 piece per line in a basic text file into an excel sheet. But sure, as long as the above criteria are met, you can do it.

Two videos

Click to expand...

NLHES said:

Thanks much for this!

Extracting all pages is all I'm trying to do with SB, I can do the rest easily.

The reason why I've struggled thus far with scraping this particular site is IP blockage - there's a limit of 200 requests daily. So I'm hoping to overcome this by SB. In terms of the actual contents, there's no JavaScript.

Anyhow, thanks much for your comment. I will look over the videos you provided!

Click to expand...

datingsitespot said:

This is awesome thanks so much !

Click to expand...