Scrape all pages of website >> Scrapebox

pokerjk · Nov 11, 2011

I'm trying to scrape all the pages of a website with Scrapebox but its not scraping all of the pages.

Google says the site has 770,000 pages.

I set the footprint to site:domain.com select all search engines and results to max (1000).

It only pulls just over 1500 results. How do I get Scrapebox to scrape ALL the 770,000 pages?

Cheers

pokerjk · Nov 11, 2011

Oops. Yes big typo in title... tired...

Kickflip · Nov 11, 2011

Could try this to see if it pull more pages

Code:

1 site:
2 site:
3 site:
4 site:
5 site:
6 site:
7 site:
8 site:
9 site:
0 site:
q site:
w site:
e site:
r site:
t site:
y site:
u site:
i site:
o site:
p site:
a site:
s site:
d site:
f site:
g site:
h site:
j site:
k site:
l site:
z site:
x site:
c site:
v site:
b site:
m site:

pokerjk · Nov 11, 2011

Kickflip said:

Could try this to see if it pull more pages

Code:

1 site:
2 site:
3 site:
4 site:
5 site:
6 site:
7 site:
8 site:
9 site:
0 site:
q site:
w site:
e site:
r site:
t site:
y site:
u site:
i site:
o site:
p site:
a site:
s site:
d site:
f site:
g site:
h site:
j site:
k site:
l site:
z site:
x site:
c site:
v site:
b site:
m site:

Thanks I got more 5600, but no where near the 770,000... :/

VaLeRyA · Nov 11, 2011

same problem
there must be some tool to extract all urls from google

HelloInsomnia · Nov 11, 2011

Your going to have to keep adding keywords to get more results...

I would look for a sitemap, you may be able to get all the pages from there.

Edit: another idea is to load up all the urls you have already and scrape for internal links in the link extractor - you may be able to do that over and over again to multiply the results.

kokoloko75 · Nov 11, 2011

Wow, 770.000 is huge !
You need to have a tons a good proxies to scrape all this from Google...

The website doesn't have a Sitemap ? (...)
URLs can not be generated manually ? Like :

Code:

http://wwwwebsite.com/page/1
http://wwwwebsite.com/page/2
http://wwwwebsite.com/page/3
...

Beny

pokerjk · Nov 12, 2011

HelloInsomnia said:
Your going to have to keep adding keywords to get more results...

I would look for a sitemap, you may be able to get all the pages from there.

Edit: another idea is to load up all the urls you have already and scrape for internal links in the link extractor - you may be able to do that over and over again to multiply the results.

No site map :/

That's what I have been doing is extracting the internal links over and over but its a little long hauled that way.

IamTURK · Nov 12, 2011

site:domain.com

or u should scrape sitemaps.

no other way.

ppcmaster · Apr 8, 2012

How about Xenu? Have you used it before? You will want to turn off scraping external links though. It works like a charm for me

d3x73r · Oct 31, 2012

ppcmaster said:
How about Xenu? Have you used it before? You will want to turn off scraping external links though. It works like a charm for me

At first thanx for the good tip ppcmaster but...

How do you take the links to scrapebox from xenu ... it try create a sitemap but its over 10 Mb so scrapebox didnt can read them.. then i try to split the sitemap but also no luck scrapebox dint load any url. is there a method with xenux to filter out files from the sitemal like jpg ? or is there a method to export the urls to a txt from xenus ??

all the best

innocent_kid · Oct 31, 2012

if u have so many google pass private proxies than only you can do it

Scrape all pages of website >> Scrapebox

pokerjk

Senior Member

pokerjk

Senior Member

Kickflip

BANNED

pokerjk

Senior Member

VaLeRyA

Regular Member

HelloInsomnia

Elite Member

kokoloko75

Elite Member

pokerjk

Senior Member

IamTURK

Regular Member

ppcmaster

Elite Member

d3x73r

Newbie

innocent_kid

Power Member

Main Menu

Marketplace

Making Money

BlackHat World