1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

tips for building own proxies db

Discussion in 'Other Languages' started by conquering_lion, Jun 17, 2013.

  1. conquering_lion

    conquering_lion Newbie

    Joined:
    Jun 1, 2013
    Messages:
    15
    Likes Received:
    3
    Hi there,

    I'm actually making an automate, let me share with you !

    I making a bot that will be able to :

    - get a proxies in a db (I found 30 000 around uniques couples proxy_ip:proxy_port , loaded in mysql db , interacting with rails app, but I use mysql gem for work with independant ruby code)
    - find another couples be searching in google and explore top 10 serps for each entries, or explore serps on specials keywords (free proxy, proxy list, etc)
    - check if proxy is valid. First condition, I go to a website that display ip of the client and tell if there is proxy detected or not (I can set my "active" and "anonymous" fields in db there), and just go to google homepage (for determine if "google_banned" field is set to true or false), although I know that you have to make search in google to see if really banned or not.

    However I have some interrogations ...

    °Proxy could be invalid at one time but free of use (not much users on it) on another time
    °same interrogation about google banned term..
    °how to success and get a valid and useable proxy quick

    I precise I have make a timer for allocate queries in time.. but I guess my time to sleep too long (35 to 75 sec) because sometimes I get 5 firsts pages of google serps an the next query fail..

    That's all for tonight.. I hope you will enjoy my reflexion and tips describe above :D
    comments welcome
     
  2. Ramirez

    Ramirez Newbie

    Joined:
    Jul 2, 2013
    Messages:
    18
    Likes Received:
    6
    I maintain my own proxy lists. One tip: do not remove a proxy from your DB the first time it is offline. Increment a counter, and only remove it on, for example, the 5th consecutive time it is offline. Public proxies go on/offline frequently.

    Another tip: host your own proxyjudge on your own server. Do not rely on public proxyjudges as the owners can detect and steal your proxies.
     
    • Thanks Thanks x 1
  3. conquering_lion

    conquering_lion Newbie

    Joined:
    Jun 1, 2013
    Messages:
    15
    Likes Received:
    3
    thanks for sharing ideas :p
     
  4. controls

    controls Newbie

    Joined:
    Jul 4, 2011
    Messages:
    47
    Likes Received:
    13
    What a great read really interesting
     
  5. Eightomic

    Eightomic Registered Member Premium Member

    Joined:
    Oct 20, 2012
    Messages:
    60
    Likes Received:
    13
    Occupation:
    Web Developer / Designer
    Location:
    United States
    Home Page:
    For maintaining a proxy database, make sure to remove dead proxies whenever possible and keep your sources fresh; 30,000 public proxies might only last you a week.