1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

How can I scrape all URLs of a website in SERPs

Discussion in 'Black Hat SEO Tools' started by night undertaker, Dec 1, 2018.

  1. night undertaker

    night undertaker BANNED BANNED

    Joined:
    May 30, 2012
    Messages:
    56
    Likes Received:
    21
    Scrapebox scrapes only a few of them, just couple hundreds. I need to scrape about 20k URLs of my client’s website. How can I do that?

    I tried to collect those URLs through website with screaming frog, but this shit also doesn’t help.

    Thanks
     
  2. mindmaster

    mindmaster Jr. VIP Jr. VIP

    Joined:
    Sep 16, 2010
    Messages:
    4,339
    Likes Received:
    2,027
    Home Page:
    You can either use scrapebox to scrape from google index. Site:website.com + a long list of generic words.
    Or you can use one of the addons like Link Extractor. Start with a few pages, scrape. Add the new pages scraped and repeat until you get most pages.

    Screaming frog can work as well, but the learning curve takes a bit longer.
     
    • Thanks Thanks x 2
  3. ThatSEO

    ThatSEO Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2016
    Messages:
    1,310
    Likes Received:
    1,038
    Gender:
    Male
    Occupation:
    Self employed marketing stuff
    Location:
    Sometimes UK
    Get better proxies or as above said, use site:domain

    Or open the xml sitemap into excel and import
     
  4. night undertaker

    night undertaker BANNED BANNED

    Joined:
    May 30, 2012
    Messages:
    56
    Likes Received:
    21
    Why the hell scrapebox can’t scrape easily?
     
  5. ThatSEO

    ThatSEO Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2016
    Messages:
    1,310
    Likes Received:
    1,038
    Gender:
    Male
    Occupation:
    Self employed marketing stuff
    Location:
    Sometimes UK

    Because you’re not not using it right
     
    • Thanks Thanks x 6
  6. night undertaker

    night undertaker BANNED BANNED

    Joined:
    May 30, 2012
    Messages:
    56
    Likes Received:
    21
    I was using it when you were first erected in shuttle bus
     
  7. ThatSEO

    ThatSEO Jr. VIP Jr. VIP

    Joined:
    Jan 22, 2016
    Messages:
    1,310
    Likes Received:
    1,038
    Gender:
    Male
    Occupation:
    Self employed marketing stuff
    Location:
    Sometimes UK

    Well learn to use it properly then
     
    • Thanks Thanks x 1
  8. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    4,931
    Likes Received:
    2,630
    Gender:
    Male
    Home Page:
    Scrapebox can scrape over a million urls per minute I have a video showing it in fact.

    It can scrape easily, but it can't circumvent blocks.

    So you could use the grab links by crawling a site function to scrape all the urls of the site or do as mind master said and tack on a bunch of keywords.

    Because the search engines hard limit to 1000 results per request and soft limit to 300 to 600 for advanced queries, often times.

    So lots of keywords forces the engine to return different sets of results and you remove duplicates when done. But unless you need to know what is indexed in google, I wouldn't bother with them, as if you just crawl the site, you don't have to worry about google banning ips.

    That said, the end site webserver may ban your ips, but that just means you need more private proxies or just go slower.

     
    • Thanks Thanks x 2
  9. Aty

    Aty Jr. VIP Jr. VIP

    Joined:
    Jan 27, 2011
    Messages:
    7,196
    Likes Received:
    4,859
    Home Page:
    Screaming Frog
     
  10. brandon999

    brandon999 Registered Member

    Joined:
    Aug 5, 2010
    Messages:
    52
    Likes Received:
    74
    you can use scrapebox link extractor on your client's website. if you are scraping specific data or all data, you can use custom grabber using the links that you scraped using link extractor.:D
     
    • Thanks Thanks x 1
  11. majento

    majento Newbie

    Joined:
    Oct 1, 2017
    Messages:
    39
    Likes Received:
    33
    Gender:
    Male
    Home Page:
    site-analyzer.pro - free seo crawler