1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping google

Discussion in 'Black Hat SEO Tools' started by emacs2, Aug 21, 2013.

  1. emacs2

    emacs2 Newbie

    Joined:
    Mar 24, 2011
    Messages:
    42
    Likes Received:
    5
    I now tried a lot of stuff and i am not sure if its my fault or its just over these days. I am scraping google with Scrapebox. So i got 10 semi dedicated proxies, used single harvester with random delay and my proxies get banned after say around 30 minutes.

    I am just searching for a way to scrape google solid. I dont need millions of urls a day i just scrape for simple stuff like "my house blue". I dont use advanced search operators.

    I am a long term SB user i bought it around 2 years ago. Do you guys have any advice or tips for me?

    Thank you
     
  2. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,290
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
    get completely dedicated proxies. With the semi-dedicated you might be playing it safe, but you don't know what the others are doing and what crazy settings they are using
     
    • Thanks Thanks x 1
  3. emacs2

    emacs2 Newbie

    Joined:
    Mar 24, 2011
    Messages:
    42
    Likes Received:
    5
    still get banned even with a random delay 50-60 seconds. if dont use proxies and add a delay of 60 seconds i get banned fast too. i am so frustrated.
    with what settings do you scrape these days?
     
  4. scrapefox

    scrapefox Power Member

    Joined:
    Dec 3, 2011
    Messages:
    692
    Likes Received:
    277
    As innozemec said, just because you're being "gentle" with the proxies, it doesn't mean that someone else isn't abusing them (they're shared after-all).

    I generally use private proxies for scraping with a ratio of around 1/10, so if I'm running with 20 proxies for example, I'll use 2 connections.
     
  5. emacs2

    emacs2 Newbie

    Joined:
    Mar 24, 2011
    Messages:
    42
    Likes Received:
    5
    hmm... i used 1 connection (1/10) with 10 proxies. i bought another service that said they are not shared.
     
  6. scrapefox

    scrapefox Power Member

    Joined:
    Dec 3, 2011
    Messages:
    692
    Likes Received:
    277
    And its working or are you.still running in to problems?

    If.it was a new source they were all definitenly google passed before you started?

    Posted via Topify using Android
     
  7. andishm

    andishm Regular Member Premium Member

    Joined:
    Jul 21, 2011
    Messages:
    362
    Likes Received:
    52
    Location:
    SEO World
    Having semi-dedicated or dedicated proxies for scrapbox i never think its a good idea to invest hundreds of dollars and have them banned and probably your proxy provider is not going to refresh your ip list until next month too :(

    So better use any proxy provider who timely scrape and provide fresh list of proxies which are anonymous and Google passed can be used properly :)
     
  8. akacash

    akacash Jr. VIP Jr. VIP

    Joined:
    Jan 16, 2010
    Messages:
    806
    Likes Received:
    575
    Location:
    The Beach, USA
    Hello, I've been working on something on the side to dramatically help with this problem. emacs2 if you'd like send me a pm and I'll add you on Skype, I'd like you to test it for me if you'd like. I'd like you to test some proxies, I already know they work and they should work much better too, but I'd like to see just how much and you're perfect to test for me if you'd like. If not no biggie either and good luck. Have a nice day.
     
  9. emacs2

    emacs2 Newbie

    Joined:
    Mar 24, 2011
    Messages:
    42
    Likes Received:
    5
    Not working

    Yes all passed

    akacash. what exactly are you talking about "something"?
    andishm, you got some recommendations?