1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

scraping is taking forever

Discussion in 'Black Hat SEO' started by firstnamelastname, Dec 30, 2015.

  1. firstnamelastname

    firstnamelastname Junior Member

    Joined:
    Jun 20, 2015
    Messages:
    184
    Likes Received:
    31
    Google is banning proxies so quick these days
    The proxies that scrapebox scrapes are basically useless, either they are banned or they get banned within less than 5 minutes.
    I have not tried private ones but I would imagine I will face the same problem.
    I am trying to scrape 40,000 keywords. How can I do that? right now I scrape a few, turn off, harvest proxies again, start scraping again, turn off, harvest proxies again, it's taking forever.
    And I am not using special keywords that trigger the filters like site: or inurl:
    I am using normal keywords
    how do you guys scrape these days?
     
  2. CrazyBuddy

    CrazyBuddy Regular Member

    Joined:
    Jun 23, 2013
    Messages:
    489
    Likes Received:
    126
    Occupation:
    Part Time IM , Full time Enjoyment
    Location:
    Earth
    Which VPS service are you using ?
     
  3. Ambitious12

    Ambitious12 Elite Member

    Joined:
    Jun 26, 2014
    Messages:
    3,097
    Likes Received:
    608
    Occupation:
    No Occupation
    Location:
    Among the Stars
    Use one VPN,you can go with HMA like VPN,and that will be perfect for changing proxies which can not ban your ip.
     
  4. Bane Bentley

    Bane Bentley Junior Member

    Joined:
    Jun 13, 2013
    Messages:
    163
    Likes Received:
    30
    Location:
    BuildMyTraffic.Net
    Home Page:
    I'm afraid there isn't a perfect option.

    Why don't you try using the cloud proxies from ScrapeBox? Those work better.
     
  5. loopline

    loopline Jr. VIP Jr. VIP

    Joined:
    Jan 25, 2009
    Messages:
    3,729
    Likes Received:
    1,995
    Gender:
    Male
    Home Page:
    Do you have to scrape google or can you use yahoo, bing or some other engine? As thats the quickest path of least resistance. I actually scrape from google and just use private proxies. You would be surprised how fast it gets done even at 1 connection. Or even at 1 connection with a delay. I started up a run last night on 1 server with 50 private proxies and a 5 second delay and scraped over 200K results in about 16 hours. I mean if you only need 40K results, grab 10 proxies, set a delay of 10 in the detailed harvester and just minimize it and come back in a couple days. Or find another engine.

    I scrape a lot of results daily, but most of them do not come from google, they come from more creative methods.
     
  6. TheSEOWizard

    TheSEOWizard Power Member

    Joined:
    Aug 20, 2011
    Messages:
    548
    Likes Received:
    156
    Occupation:
    SEO, PBN, Website et all
    Location:
    PBN World/SEO Land
    Google bans a lot of proxies nowadys. Even if you have clean Google passed proxies, you need to ratio very less i.e. around 1 thread per 10 proxies. Or you might try the scanned proxies service of some popular seller such as proxygo to get some good Google passed proxies everyday. And as loopline mentioned, there are other search engines too except Google and you just need to be a bit more creative. Also make sure to update the search engines in Scrapebox by downloading automatically from their servers as there were updates released a couple of times in the last few months or so.
     
  7. apex1

    apex1 Junior Member

    Joined:
    May 29, 2015
    Messages:
    112
    Likes Received:
    98
    Get private proxies

    Fix your scraper settings
     
  8. Google Prince

    Google Prince Jr. VIP Jr. VIP

    Joined:
    Dec 24, 2015
    Messages:
    158
    Likes Received:
    89
    Location:
    Google's Search Engine
    I've been using scrapebox all day with no problems.First and foremost public proxies are horrible and I would invest in private proxies. I recommend buyproxies as I use them with great success and scrape for hours. GL
     
  9. PaulSmithAUS

    PaulSmithAUS Registered Member

    Joined:
    Nov 16, 2015
    Messages:
    99
    Likes Received:
    38
    Free proxies are free for a reason. Use private proxies and set your settings so they don't look like your a scraper.