1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

google/yahoo scraping limits

Discussion in 'Black Hat SEO' started by HaxTuTorial, Jun 22, 2013.

  1. HaxTuTorial

    HaxTuTorial Jr. VIP Jr. VIP

    Joined:
    Mar 13, 2010
    Messages:
    425
    Likes Received:
    127
    Home Page:
  2. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,288
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
  3. cooltoad

    cooltoad Senior Member

    Joined:
    Sep 10, 2010
    Messages:
    934
    Likes Received:
    549
    Occupation:
    None of your business
    Location:
    On Vacation
    Sure can! The proxies will have to be private and you should probably do away with the Multi Threaded Harvester in SB.

    cheers
    CT
     
  4. hpv222

    hpv222 Power Member

    Joined:
    Feb 8, 2010
    Messages:
    736
    Likes Received:
    274
    Not with Google unless you get tons of proxies; they temp-ban everything left and right; some of the private proxies that I use are perm-banned as well, make sure you ask the proxy provider before buying. Yahoo and Bing are more relaxed, so far I haven't seen them banning an IP, but I've never run more than several dozen queries per day either.
     
  5. cooltoad

    cooltoad Senior Member

    Joined:
    Sep 10, 2010
    Messages:
    934
    Likes Received:
    549
    Occupation:
    None of your business
    Location:
    On Vacation
    Woah! Temp ban of a private proxy - That's just so unfair :p
    Anyways, this can be achieved, but you will have to gather up a considerable amount of proxies.
    As far as Bing as goes, it just responds so slow while scraping!

    cheers
    CT
     
  6. HaxTuTorial

    HaxTuTorial Jr. VIP Jr. VIP

    Joined:
    Mar 13, 2010
    Messages:
    425
    Likes Received:
    127
    Home Page:
    wow really 100k per day? wow how much proxies would I be looking at if I wanted 100k per day on yahoo/bing? 1000?
     
  7. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,853
    Likes Received:
    7,457
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    Recently I got blocked by Bing with just a few hundred queries. I didn't think they'd ban me so I just went ahead without proxies. Turns out they're keeping a better watch automated queries.
     
  8. HaxTuTorial

    HaxTuTorial Jr. VIP Jr. VIP

    Joined:
    Mar 13, 2010
    Messages:
    425
    Likes Received:
    127
    Home Page:
    Maybe u did too many per hour? I've been testing bing and yahoo's ban limits to see when a ban occurs, i'm still in the process of testing them.
     
  9. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,853
    Likes Received:
    7,457
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    Yeah I didn't think they'd block me but they did, so I didn't throttle. My broadband is not stellar either and the connection is shared at work so it must have been a few hundred requests in 5 to 10 minutes max.