how to scrape only root folder urls?

[alwayslistening]

using scrapebox, whats the best way to pull only the urls within a directory without crawling the entire site?

Example: xyz.com/users

looking to grab only the site links like:

xyz.com/users/mike
xyz.com/users/dan

 

[loopline]

using scrapebox, whats the best way to pull only the urls within a directory without crawling the entire site?

Example: xyz.com/users

looking to grab only the site links like:

xyz.com/users/mike
xyz.com/users/dan

You can do

site:xyz.com/users/

in google, but thats really it. That or let scrapebox crawl the whole site and just remove all urls not containing xyz.com/users/

 

[BlackShield]

using scrapebox, whats the best way to pull only the urls within a directory without crawling the entire site?

Example: xyz.com/users

looking to grab only the site links like:

xyz.com/users/mike
xyz.com/users/dan

You can use Google to extract the exact pages that you need by using operators.

Use site:xyz.com/users to extract the pages which are all under the users page.
You can also use inurl:users site:xyz.com to extract the pages that you required.

 

[alwayslistening]

You can do

site:xyz.com/users/

in google, but thats really it. That or let scrapebox crawl the whole site and just remove all urls not containing xyz.com/users/

Yeah sorry was late, stupid question. Google would be limited. Ok so I found the search string and somehow doesn’t seem like it limits numbers on the site meaning I can go to page 5,000 and it’s still showing results.


How would I go about duplicating a search string while adding a new number in each new duplicate?


Xyz.com/users/search/1

Now I need this to be duplicated all the way up to whatever number but not sure how to do so. From there I can scrape each profile link/export/ then rescrape that list for email on profile which is what’s desired. Thanks.

1) Use site:xyz.com/users to extract the pages which are all under the users page.


2) You can also use inurl:users site:xyz.com to extract the pages that you required.

1) this doesn’t work
2) google will only show 100 pages worth of results I believe

 

[Zwielicht]

Thread Moved​

 

[bartosimpsonio]

You can use wget and pass the -np command line option, it won't go into deeper directories.

 

[alwayslistening]

You can use wget and pass the -np command line option, it won't go into deeper directories.

can you explain more in depth? If not no problem might be over my head

 

[bartosimpsonio]

can you explain more in depth? If not no problem might be over my head

wget --mirror -np hxxp://xyz.com/users

 

[loopline]

Yeah sorry was late, stupid question. Google would be limited. Ok so I found the search string and somehow doesn’t seem like it limits numbers on the site meaning I can go to page 5,000 and it’s still showing results.


How would I go about duplicating a search string while adding a new number in each new duplicate?


Xyz.com/users/search/1

Now I need this to be duplicated all the way up to whatever number but not sure how to do so. From there I can scrape each profile link/export/ then rescrape that list for email on profile which is what’s desired. Thanks.



1) this doesn’t work
2) google will only show 100 pages worth of results I believe

For the google part you can tack on letters and numbers like
site:xyz.com/users/ 1
site:xyz.com/users/ 2
site:xyz.com/users/ a
site:xyz.com/users/ b
site:xyz.com/users/ c
site:xyz.com/users/ purple
site:xyz.com/users/ car
etc..

That forces google return different sets of results from its index and then just remove duplicates when your done.




For your search approach

Xyz.com/users/search/1

You just use the merge to merge in a number list so like

1
2
3
4
5
6
etc...

Online tools can make you a list easy. just google

Then use the scrapebo merge feature. so do like this and put this in a text file

Xyz.com/users/search/%kw%

Then put your numbers in the keyword box in scrapebox and hit the M merge button.

More info
http://scrapeboxfaq.com/how-do-i-use-tokens-with-the-m-merge-option

Although I felt hazy on understanding this part of the question so I might be totally off base in what your trying to do.

 

[alwayslistening]

For the google part you can tack on letters and numbers like


For your search approach

Xyz.com/users/search/1

You just use the merge to merge in a number list so like

1
2
3
4
5
6
etc...

Online tools can make you a list easy. just google


Although I felt hazy on understanding this part of the question so I might be totally off base in what your trying to do.

thanks loopline, actually i just needed to take xyz.com/search=1 and duplicate that all the way up to 50,000 or something. Ive tried never can figure it out. ill just crawl those search queries and go level one to grab profile url... then ill rescrape for email/url off their profiles.

So now i found this which is part one of the solution:

https://textmechanic.com/text-tools/numeration-tools/generate-list-numbers/

great can export as txt....

now i just learned the merge feature thanks to you. This is actually something i needed to learn as with many sites clearly you need to scrape the search pages and i couldent figure out how to mass produce urls with numbers but now i have it. thanks a lot.

 

[loopline]

thanks loopline, actually i just needed to take xyz.com/search=1 and duplicate that all the way up to 50,000 or something. Ive tried never can figure it out. ill just crawl those search queries and go level one to grab profile url... then ill rescrape for email/url off their profiles.

So now i found this which is part one of the solution:

https://textmechanic.com/text-tools/numeration-tools/generate-list-numbers/

great can export as txt....

now i just learned the merge feature thanks to you. This is actually something i needed to learn as with many sites clearly you need to scrape the search pages and i couldent figure out how to mass produce urls with numbers but now i have it. thanks a lot.

Your welcome. Also if you have the email scraper plugin, it has a url generation tab, that has some handy features.