1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

scraping is taking forever

Discussion in 'Black Hat SEO' started by firstnamelastname, Dec 30, 2015.

  1. firstnamelastname

    firstnamelastname Junior Member

    Joined:
    Jun 20, 2015
    Messages:
    197
    Likes Received:
    32
    Google is banning proxies so quick these days
    The proxies that scrapebox scrapes are basically useless, either they are banned or they get banned within less than 5 minutes.
    I have not tried private ones but I would imagine I will face the same problem.
    I am trying to scrape 40,000 keywords. How can I do that? right now I scrape a few, turn off, harvest proxies again, start scraping again, turn off, harvest proxies again, it's taking forever.
    And I am not using special keywords that trigger the filters like site: or inurl:
    I am using normal keywords
    how do you guys scrape these days?
     
  2. CrazyBuddy

    CrazyBuddy Regular Member

    Joined:
    Jun 23, 2013
    Messages:
    491
    Likes Received:
    126
    Occupation:
    Part Time IM , Full time Enjoyment
    Location:
    Earth
    Which VPS service are you using ?
     
  3. Ambitious12

    Ambitious12 Elite Member

    Joined:
    Jun 26, 2014
    Messages:
    3,096
    Likes Received:
    609
    Occupation:
    No Occupation
    Location:
    Among the Stars
    Use one VPN,you can go with HMA like VPN,and that will be perfect for changing proxies which can not ban your ip.
     
  4. Bane Bentley

    Bane Bentley Jr. VIP Jr. VIP

    Joined:
    Jun 13, 2013
    Messages:
    180
    Likes Received:
    35
    I'm afraid there isn't a perfect option.

    Why don't you try using the cloud proxies from ScrapeBox? Those work better.
     
  5. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,807
    Likes Received:
    2,027
    Gender:
    Male
    Home Page:
    Do you have to scrape google or can you use yahoo, bing or some other engine? As thats the quickest path of least resistance. I actually scrape from google and just use private proxies. You would be surprised how fast it gets done even at 1 connection. Or even at 1 connection with a delay. I started up a run last night on 1 server with 50 private proxies and a 5 second delay and scraped over 200K results in about 16 hours. I mean if you only need 40K results, grab 10 proxies, set a delay of 10 in the detailed harvester and just minimize it and come back in a couple days. Or find another engine.

    I scrape a lot of results daily, but most of them do not come from google, they come from more creative methods.
     
  6. TheSEOWizard

    TheSEOWizard Power Member

    Joined:
    Aug 20, 2011
    Messages:
    548
    Likes Received:
    156
    Occupation:
    SEO, PBN, Website et all
    Location:
    PBN World/SEO Land
    Google bans a lot of proxies nowadys. Even if you have clean Google passed proxies, you need to ratio very less i.e. around 1 thread per 10 proxies. Or you might try the scanned proxies service of some popular seller such as proxygo to get some good Google passed proxies everyday. And as loopline mentioned, there are other search engines too except Google and you just need to be a bit more creative. Also make sure to update the search engines in Scrapebox by downloading automatically from their servers as there were updates released a couple of times in the last few months or so.
     
  7. apex1

    apex1 Junior Member

    Joined:
    May 29, 2015
    Messages:
    174
    Likes Received:
    149
    Get private proxies

    Fix your scraper settings
     
  8. Google Prince

    Google Prince Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2015
    Messages:
    162
    Likes Received:
    93
    Location:
    Google's Search Engine
    I've been using scrapebox all day with no problems.First and foremost public proxies are horrible and I would invest in private proxies. I recommend buyproxies as I use them with great success and scrape for hours. GL
     
  9. PaulSmithAUS

    PaulSmithAUS Registered Member

    Joined:
    Nov 16, 2015
    Messages:
    99
    Likes Received:
    38
    Free proxies are free for a reason. Use private proxies and set your settings so they don't look like your a scraper.