1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HTTP/SOCKS Proxies for (legal) screen scrapers

Discussion in 'Proxies' started by Baybo.it, Aug 9, 2011.

Tags:
  1. Baybo.it

    Baybo.it Registered Member

    Joined:
    Aug 9, 2011
    Messages:
    72
    Likes Received:
    39
    Occupation:
    Founder of Baybo.it
    Location:
    San Francisco
    Home Page:
    We were running a little (legal) screen scraper and had to do it behind an HTTP proxy. I found it difficult to find a good, curated list of HTTP and SOCKS proxies.

    We would just run in the term:
    Code:
    export http_proxy=foo.bar.baz.qux
    And iterate through a list of proxies taken from these sites (which tend to work well):
    • sockslist
    • proxyhttp