1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Having Problems Scraping with Proxies

Discussion in 'PHP & Perl' started by sevenfigurez, Sep 21, 2012.

  1. sevenfigurez

    sevenfigurez Junior Member

    Joined:
    Nov 5, 2009
    Messages:
    183
    Likes Received:
    46
    I built a script using curl that scrapes google no problem. Once I implement the proxies, I immediately get hit with "Sorry, but your computer or network may be sending automated queries". Makes no sense...I can use VPN, change proxies, and re run script, but when I try to use proxies to run it it gets blocked.

    What can I be missing?
     
  2. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,143
    You need other proxies :)
     
  3. cgimaster

    cgimaster Power Member

    Joined:
    Jun 30, 2012
    Messages:
    525
    Likes Received:
    311
    Gender:
    Male
    Are you saving and reusing your cookies ?
    Are you passing back the verifications calls from page to page ?

    There are several factors that identify your calls as a BOT to ban you really fast.

    Also make sure you have a clean list of proxies most public ones are already blacklisted and will present the message you mentioned
     
  4. kilaz

    kilaz Jr. VIP Jr. VIP

    Joined:
    Aug 15, 2009
    Messages:
    875
    Likes Received:
    382
    Home Page:
    I do the same thing.. Some things you need to make sure of.

    Make sure your using anonymous proxies.
    stay away from public proxies that have been abused and not passing google traffic by investing in a good proxy list, Exclusive proxy list, Shared proxies or Private proxies.

    What I do in my script is first test the proxies to see if they are working and passing google, Then rotate a proxy for each search. Seems to work great.