1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

what is the best latency for the free proxies harvested?

Discussion in 'Proxies' started by caridead, Dec 18, 2010.

  1. caridead

    caridead Newbie

    Joined:
    Oct 5, 2009
    Messages:
    40
    Likes Received:
    6
    Home Page:
    someone mind to share that how to choose/filter the good proxy for scrapping?

    what is the level of proxy latency that good for us?
     
  2. Kickflip

    Kickflip BANNED BANNED

    Joined:
    Jan 29, 2010
    Messages:
    2,038
    Likes Received:
    2,465
    I just use anything under 3000ms for scraping. Under 1500ms for posting.
     
  3. SpeakToTman

    SpeakToTman Regular Member

    Joined:
    May 31, 2009
    Messages:
    249
    Likes Received:
    158
    Location:
    BHW
    Home Page:
    Similar...i try not go above about 2000ms
     
  4. rever20

    rever20 Junior Member

    Joined:
    Jan 11, 2010
    Messages:
    144
    Likes Received:
    21
    No more than 3000 IMO.
     
  5. caridead

    caridead Newbie

    Joined:
    Oct 5, 2009
    Messages:
    40
    Likes Received:
    6
    Home Page:
    but the worst case i was using below 2000ms also cause the posting time out.... needed to use the premium proxy...

    Any free ideas to solve the time out issue?
     
  6. cooooookies

    cooooookies Senior Member

    Joined:
    Oct 6, 2008
    Messages:
    1,008
    Likes Received:
    216
    Actually, the latency does not necesarilly matter. You are running your apps multithreaded one the one or other thread needs more time to finish. If you have many slow proxies, just increase the number of threads until you reach your CPU or bandwidth capacity.

    However, reponse times > 5000 ms (just to mention a number, this is certainly not applicable for all proxies but a good rule of thumb for me) mean in the tendency that the proxy is overcrowded and the higher the response time, the higher the probability is that your web connection gets cut. If you have multiple GET/POST requests which need to be carried out sequentially, then you are in trouble.
     
    Last edited: Dec 18, 2010