1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Working on proxy bot. What features would you like?

Discussion in 'Proxies' started by sknight, Jan 14, 2011.

  1. sknight

    sknight Newbie

    Joined:
    Nov 26, 2010
    Messages:
    32
    Likes Received:
    11
    I wrote a proxy bot during Xmas. It is running on a high end 4 CPU box as many daemons. It scrapes the top 300 serp pages on google, bing, yahoo over an hour as well as hits 20 daily proxy list sites and the hardcoded scapebox proxies lists 4 times a day accumlating open proxies. It has 60 processes that are constantly checking the found proxy databases for updating.

    So far since xmas it has discovered and tried 170,000+ proxies, of these it found 30,000+ of them active. As I write this message, it currrently has 1,588 proxies that are fast and alive. Working proxies found are removed due to being becoing unavailable or not responding back to a test loop within 3 seconds (I don't want slow overloaded proxies).

    It also scans the headers in the loop test for any reference to the requesting ip address, requesting host name or any _X_ parameters or VIA parameters. It has a dump proxy list feature for scapebox and since it checks scrapeboxes hardcoded proxies sites, I only have to load and check this one proxy list. The list also works with firefox's premium proxy switcher plugin.

    I will be adding:
    . check entire class C range of found proxies
    . check against spam stopper engines

    What I am asking the bhw community for is:

    . known spam stopper sites for the bot to check against, if anyone cares to share
    . more daily proxy listing sites for the bot to scan, if anyone cares to share
    . more features people would like to see

    Thank you for your time! I and hoping to get the bot out of alpha and into beta with the next week
     
  2. unknwn

    unknwn BANNED BANNED

    Joined:
    Dec 19, 2009
    Messages:
    151
    Likes Received:
    59
    There is already a excellent tool that does everything you had in mind and much more and is even free http://www.proxyfire.net/forum/showthread.php?t=33
     
  3. sknight

    sknight Newbie

    Joined:
    Nov 26, 2010
    Messages:
    32
    Likes Received:
    11
    Well, isn't that a cool program. Bit of a resource hog and it requires my attention on my machine but a very valuable resource (and the web site too!)

    Thanx!
     
  4. sknight

    sknight Newbie

    Joined:
    Nov 26, 2010
    Messages:
    32
    Likes Received:
    11
    I'm surprised no one has commented.

    New feature added: log into a forum and scrape messages.