1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Using Tons of Search Operator while scrapping = REALLY quick ban?

Discussion in 'Black Hat SEO' started by J0kerz, Jan 17, 2011.

  1. J0kerz

    J0kerz Supreme Member

    Joined:
    Nov 2, 2009
    Messages:
    1,415
    Likes Received:
    435
    Occupation:
    IM
    Location:
    There
    Hey there,

    I am currently scrapping Google using 3-4 search operators (inurl,intext,intitle,etc..) in each query.

    The Proxies that I am using seems to get banned way faster if I compare to when I use simple query with few operators in each query (0-1).

    Is there anyway to avoid such a quick ban?

    I CANNOT harvest urls from Google when I try to use really optimized scrapping footprint.
     
  2. roberteb

    roberteb Regular Member

    Joined:
    Oct 30, 2010
    Messages:
    402
    Likes Received:
    120
    Location:
    UK
    I found the same thing the more complex the search string the quicker the ban. I only found it to be a problem on my desktop using a single IP. When scraping using SB and a large list of public proxies I rarely have a problem with it.
     
  3. andreyg13

    andreyg13 Jr. VIP Jr. VIP

    Joined:
    Nov 13, 2009
    Messages:
    915
    Likes Received:
    1,774
    Occupation:
    SEO
    Location:
    http://seoshark.org
    Home Page:
    It is best to use one at a time, google constatntly changes things and i have seen this even with one search operator searching something just randomly, not even scrapping. I had to enter captcha like 6 times until i got to page 20.
     
  4. Bartholomew

    Bartholomew Regular Member

    Joined:
    Dec 31, 2009
    Messages:
    290
    Likes Received:
    103
    Home Page:
    Google doesn't like seo queries, that's a fact. If possible use another footprints, like instead of "inurl:forum inurl:index.php" you might try "In total there are users online Most users ever online". If there's no such footprint try to program your scrapper to search for something other, not related to seo, every few queries to water down your stream of queries.
     
  5. J0kerz

    J0kerz Supreme Member

    Joined:
    Nov 2, 2009
    Messages:
    1,415
    Likes Received:
    435
    Occupation:
    IM
    Location:
    There
    I find it very annoying since the only way to find Great urls is to use complex footprints.

    I guess I could use less private proxies with a sleep time in between each queries.
     
  6. jb2008

    jb2008 Senior Member

    Joined:
    Jul 15, 2010
    Messages:
    1,158
    Likes Received:
    972
    Occupation:
    Scraping, Harvesting in the Corn Fields
    Location:
    On my VPS servers
    Try using in quotes instead of a special operator. If the inurl / intext query is unique enough it will be found in the url anyway. For example if you search "/article.php?news33/module/bhw" it will be found in the URL anyway 99% of the time. It's the only way to harvest large amounts of keywords. Special operators are nice for quick harvests but you will need like 10,000 working proxies to do a big harvest. Not realistic. I use quotes instead of URLs and have no problems. If your footprint in quotes is still not getting you what you want then you must tweak it until it does.
     
  7. J0kerz

    J0kerz Supreme Member

    Joined:
    Nov 2, 2009
    Messages:
    1,415
    Likes Received:
    435
    Occupation:
    IM
    Location:
    There
    Test successful! :)

    I am currently scrapping using 50 Private proxies on 50 Thread, with 60 seconds between each query and it seems to work like a charm!
     
    • Thanks Thanks x 1