1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping from Google...Private or Public?

Discussion in 'Proxies' started by crackaway, Dec 24, 2011.

  1. crackaway

    crackaway Junior Member

    Joined:
    Jun 8, 2011
    Messages:
    114
    Likes Received:
    8
    I have a tool that scrapes specific information from Google. My server was already banned/blocked by them so we've rewritten the tool to be able to use proxies. What is the best way to do this out getting blocked again? What we do is scrap the information and then cache it. We rescrap/update the information every week or so. Should I spend money on something like squidproxy or use the $5 public proxy offerred from thebigproxylist.com? Your thoughts?
     
  2. Execute

    Execute Supreme Member

    Joined:
    Aug 30, 2010
    Messages:
    1,349
    Likes Received:
    5,017
    Location:
    United Kingdom
    What I usually do is scrape data with public proxies that I personally scrape with software that I own. Then when it comes to posting I use private proxies to speed up the process.

    Of course you can use private proxies to scrape with but I do not think it is needed.

    Hope that helped :)
     
    • Thanks Thanks x 1
  3. crackaway

    crackaway Junior Member

    Joined:
    Jun 8, 2011
    Messages:
    114
    Likes Received:
    8
    I guess I need to shed some more light on my intentions. I scrape Google by using a wordpress plugin. the plugin will pull information from a source (one of google's services) and then display it on the site. the information I pull is geographically specific. meaning the info I pull in US will be slightly different in another country (currently, listing may be in a different language, etc). Though I have users from all over the world, it would be hard to cater to each ones location. I think the best thing to do is go with the majority. The majority of my users are US based so I should use US proxies to pull the information. i am not worried what proxy I use to display the information, I think all it matters is what proxy pulls the information. So I guess my question is, where can I get really really fast US proxies from?