1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Better to go with multiple servers or a proxy system?

Discussion in 'Black Hat SEO Tools' started by Viltedali, Dec 24, 2012.

  1. Viltedali

    Viltedali Regular Member

    Joined:
    Feb 10, 2008
    Messages:
    305
    Likes Received:
    32
    Location:
    Midwest-US
    I need to monitor/scrape search results on a web page. Apparently the cron was set to run too frequently, and the IP address of my server was blocked.

    I uploaded to a different server and it was not blocked, so I guess they blocked the IP address only.

    My question is, would it be better to use one server (a VPS) and some sort of proxy system, or use multiple shared servers to continue scraping the webpage.

    Free proxies too often show your true IP address, yes?

    A VPS I guess would cost me about $50 a month, what would I have to pay for say 10 private proxies per month? What would I have to pay for code that would use the proxies in rotation each time the script runs, the script is written in PHP.

    I'm guessing you can still get shared hosting accounts for about $5 a month.

    Thanks in advance for any replies.
     
  2. Autumn

    Autumn Elite Member

    Joined:
    Nov 18, 2010
    Messages:
    2,197
    Likes Received:
    3,041
    Occupation:
    I figure out ways to make money online and then au
    Location:
    Spamville
    Private proxies for sure because:

    a) it's much cheaper to get new IPs (private proxies are about $2 or less per month each)
    b) it's easy to get your proxy provider to swap to new IPs if your old ones get banned
    c) proxy providers are usually much better about protecting your anonymity
    d) you can run your script and proxies from one location, rather than having the same script running on multiple virtual accounts or VPSs and having to combine your results etc.

    Free proxies are good for some instances but I wouldn't use them for something that you had to scrape reliably on a regular basis.
     
    • Thanks Thanks x 2
  3. TheRealRazzy

    TheRealRazzy Jr. VIP Jr. VIP

    Joined:
    Mar 2, 2011
    Messages:
    1,464
    Likes Received:
    1,761
    Location:
    Chicago, IL
    Home Page:
    I agree 100% with Autumn.

    I was just going to tell you that it would be much cheaper but she went into super detail :D
     
    • Thanks Thanks x 1
  4. s3v007

    s3v007 Junior Member

    Joined:
    Jun 25, 2012
    Messages:
    104
    Likes Received:
    29
    Location:
    The Internet
    Just buy a bunch of proxies and have your scraper shuffle through them. It's much cheaper and easier than having multiple servers.

    Happy Holidays!

    - s3v
     
    • Thanks Thanks x 1
  5. Viltedali

    Viltedali Regular Member

    Joined:
    Feb 10, 2008
    Messages:
    305
    Likes Received:
    32
    Location:
    Midwest-US
    Thanks for the replies.

    Another thing I have to consider is the load on the server.

    Can someone point me to a script that would rotate the proxies, access the site through the proxies, however you explain it, that I could integrate into the PHP script?

    Also, can someone point me to a good proxy provider? If I wanted 10 private proxies, I would be looking at $20 per month?
     
  6. Viltedali

    Viltedali Regular Member

    Joined:
    Feb 10, 2008
    Messages:
    305
    Likes Received:
    32
    Location:
    Midwest-US
    Is this what I would need:

    PHP:
    http://stackoverflow.com/questions/4888667/php-and-rotating-proxies/

    It looks like the IP addresses would be chosen randomly, is there a way to make it so that it would just go down the line and choose them in order?

    Thanks again for any replies.
     
  7. Autumn

    Autumn Elite Member

    Joined:
    Nov 18, 2010
    Messages:
    2,197
    Likes Received:
    3,041
    Occupation:
    I figure out ways to make money online and then au
    Location:
    Spamville
    You could do it that way, but that's overkill in my opinion. Easier way is to put all your proxies in a text file, load your proxies into an array from that text file, and then iterate through your list, adding the proxies directly into your curl/php script with curl_setopt($ch, CURLOPT_PROXY, $proxy_host:$proxy_port). That's a $10-20 job for someone on the BHW freelance board. That is of course assuming that you're using curl in your PHP script already.

    For proxy providers check out the proxy providers forum here, and read the reviews.
     
    • Thanks Thanks x 2