how to scrape only root folder urls?

using scrapebox, whats the best way to pull only the urls within a directory without crawling the entire site?

Example: xyz.com/users

looking to grab only the site links like:

xyz.com/users/mike
xyz.com/users/dan

alwayslistening said:

using scrapebox, whats the best way to pull only the urls within a directory without crawling the entire site?

Example: xyz.com/users

looking to grab only the site links like:

xyz.com/users/mike
xyz.com/users/dan

Click to expand...

You can do

site:xyz.com/users/

in google, but thats really it. That or let scrapebox crawl the whole site and just remove all urls not containing xyz.com/users/

alwayslistening said:

using scrapebox, whats the best way to pull only the urls within a directory without crawling the entire site?

Example: xyz.com/users

looking to grab only the site links like:

xyz.com/users/mike
xyz.com/users/dan

Click to expand...

You can use Google to extract the exact pages that you need by using operators.

Use site:xyz.com/users to extract the pages which are all under the users page.
You can also use inurl:users site:xyz.com to extract the pages that you required.

loopline said:

You can do

site:xyz.com/users/

in google, but thats really it. That or let scrapebox crawl the whole site and just remove all urls not containing xyz.com/users/

Click to expand...

Yeah sorry was late, stupid question. Google would be limited. Ok so I found the search string and somehow doesn’t seem like it limits numbers on the site meaning I can go to page 5,000 and it’s still showing results.


How would I go about duplicating a search string while adding a new number in each new duplicate?


Xyz.com/users/search/1

Now I need this to be duplicated all the way up to whatever number but not sure how to do so. From there I can scrape each profile link/export/ then rescrape that list for email on profile which is what’s desired. Thanks.

Aiden_1 said:

1) Use site:xyz.com/users to extract the pages which are all under the users page.


2) You can also use inurl:users site:xyz.com to extract the pages that you required.

Click to expand...

1) this doesn’t work
2) google will only show 100 pages worth of results I believe

You can use wget and pass the -np command line option, it won't go into deeper directories.

bartosimpsonio said:

You can use wget and pass the -np command line option, it won't go into deeper directories.

Click to expand...

can you explain more in depth? If not no problem might be over my head

alwayslistening said:

can you explain more in depth? If not no problem might be over my head

Click to expand...

wget --mirror -np hxxp://xyz.com/users

alwayslistening said:

Yeah sorry was late, stupid question. Google would be limited. Ok so I found the search string and somehow doesn’t seem like it limits numbers on the site meaning I can go to page 5,000 and it’s still showing results.


How would I go about duplicating a search string while adding a new number in each new duplicate?


Xyz.com/users/search/1

Now I need this to be duplicated all the way up to whatever number but not sure how to do so. From there I can scrape each profile link/export/ then rescrape that list for email on profile which is what’s desired. Thanks.



1) this doesn’t work
2) google will only show 100 pages worth of results I believe

Click to expand...

For the google part you can tack on letters and numbers like
site:xyz.com/users/ 1
site:xyz.com/users/ 2
site:xyz.com/users/ a
site:xyz.com/users/ b
site:xyz.com/users/ c
site:xyz.com/users/ purple
site:xyz.com/users/ car
etc..

That forces google return different sets of results from its index and then just remove duplicates when your done.




For your search approach

Xyz.com/users/search/1

You just use the merge to merge in a number list so like

1
2
3
4
5
6
etc...

Online tools can make you a list easy. just google

Then use the scrapebo merge feature. so do like this and put this in a text file

Xyz.com/users/search/%kw%

Then put your numbers in the keyword box in scrapebox and hit the M merge button.

More info
http://scrapeboxfaq.com/how-do-i-use-tokens-with-the-m-merge-option

Although I felt hazy on understanding this part of the question so I might be totally off base in what your trying to do.

loopline said:

For the google part you can tack on letters and numbers like


For your search approach

Xyz.com/users/search/1

You just use the merge to merge in a number list so like

1
2
3
4
5
6
etc...

Online tools can make you a list easy. just google


Although I felt hazy on understanding this part of the question so I might be totally off base in what your trying to do.

Click to expand...

thanks loopline, actually i just needed to take xyz.com/search=1 and duplicate that all the way up to 50,000 or something. Ive tried never can figure it out. ill just crawl those search queries and go level one to grab profile url... then ill rescrape for email/url off their profiles.

So now i found this which is part one of the solution:

https://textmechanic.com/text-tools/numeration-tools/generate-list-numbers/

great can export as txt....

now i just learned the merge feature thanks to you. This is actually something i needed to learn as with many sites clearly you need to scrape the search pages and i couldent figure out how to mass produce urls with numbers but now i have it. thanks a lot.

alwayslistening said:

thanks loopline, actually i just needed to take xyz.com/search=1 and duplicate that all the way up to 50,000 or something. Ive tried never can figure it out. ill just crawl those search queries and go level one to grab profile url... then ill rescrape for email/url off their profiles.

So now i found this which is part one of the solution:

https://textmechanic.com/text-tools/numeration-tools/generate-list-numbers/

great can export as txt....

now i just learned the merge feature thanks to you. This is actually something i needed to learn as with many sites clearly you need to scrape the search pages and i couldent figure out how to mass produce urls with numbers but now i have it. thanks a lot.

Click to expand...

Your welcome. Also if you have the email scraper plugin, it has a url generation tab, that has some handy features.