1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How many proxies for scraping 100 of 1000s urls ?

Discussion in 'Black Hat SEO' started by Nuker, Jun 17, 2013.

  1. Nuker

    Nuker Regular Member

    Joined:
    Jan 21, 2011
    Messages:
    352
    Likes Received:
    32
    hi,

    i don't understand how people scrape 100 of 1000s urls
    i tried with 10 proxies but got blocked after getting 1000 url
    is there any settings or trick ?
     
  2. Nuker

    Nuker Regular Member

    Joined:
    Jan 21, 2011
    Messages:
    352
    Likes Received:
    32
    is this a difficult question to answer?
     
  3. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,290
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
  4. Nuker

    Nuker Regular Member

    Joined:
    Jan 21, 2011
    Messages:
    352
    Likes Received:
    32
    i used 10 thread with 10 tested proxies
     
  5. alaltaierii

    alaltaierii Supreme Member

    Joined:
    Jun 11, 2010
    Messages:
    1,408
    Likes Received:
    349
    If you are using public proxies try to get more than 10 if you want to get more results. Minimum 50.

    If you are using 10 private proxies make sure that you use only 1-2 threads. Using more threads will get your private proxies temp banned by google.
     
  6. ZDooX-2K

    ZDooX-2K BANNED BANNED

    Joined:
    Jun 17, 2013
    Messages:
    141
    Likes Received:
    30
    why is that hard?

    i can scrape millions of links...without proxies..don`t know what kind of links you looking for,..
    the programm i have if put, let`s say series of keywords, long keywords..i can let it run for 24, and scrape millions of links..
     
  7. Nuker

    Nuker Regular Member

    Joined:
    Jan 21, 2011
    Messages:
    352
    Likes Received:
    32
    what program ? and how can you scrape million without getting banned
     
  8. ZDooX-2K

    ZDooX-2K BANNED BANNED

    Joined:
    Jun 17, 2013
    Messages:
    141
    Likes Received:
    30
    This one here..
    postimg. org/4qfaxjpmb/2013_06_18_001541.png
     
  9. ZDooX-2K

    ZDooX-2K BANNED BANNED

    Joined:
    Jun 17, 2013
    Messages:
    141
    Likes Received:
    30
    s10.postimg. org/mlaycsyop/2013_06_18_002016.png

    and now i type 11k+
     
  10. mmaniac

    mmaniac Newbie

    Joined:
    Dec 17, 2009
    Messages:
    29
    Likes Received:
    8
    Occupation:
    web designer/developer
    Location:
    Internet
    that's too much. Try two threads if you have 10 proxies, that works best for me. I am talking about private proxies. This way you can scrape non stop and g00gle won't ban proxies. If your software has support for random search intervals enable that setting so it looks more natural.
     
  11. wasuki

    wasuki Regular Member

    Joined:
    Jul 10, 2010
    Messages:
    266
    Likes Received:
    45
    Occupation:
    Blogger and offsite optimization expert
    Location:
    Houston
    Home Page:
    With private proxies works best if you 2:10 ratio of threads and proxies. you wont have problem of getting banned by Google. In case of public proxies you would have to use more than 50 proxies to get the best results. Based on your requirements you can try different combinations and see which works best for you.
     
  12. Nuker

    Nuker Regular Member

    Joined:
    Jan 21, 2011
    Messages:
    352
    Likes Received:
    32
    i have scrapebox but can't get that amount of links
     
  13. partymarty4870

    partymarty4870 Elite Member

    Joined:
    Jul 7, 2010
    Messages:
    2,034
    Likes Received:
    1,690
    Location:
    I come from a land downunder
    on any big scrapes or similar I run 2 instances of SB, one doing the scrapes and one hunting for new proxies. In fact right now I'm scanning a list of 225000 drops for PR in lots of about 5000 urls. The second instance of SB is giving me about 5/600 new google passed proxies in the time it takes me to scan 20K pages for PR. Then I just drop the new ones in the list.
     
  14. Nuker

    Nuker Regular Member

    Joined:
    Jan 21, 2011
    Messages:
    352
    Likes Received:
    32
    i just want to understand how so scrape without limit using proxies ?
     
  15. queenseo

    queenseo Newbie

    Joined:
    Jul 3, 2013
    Messages:
    38
    Likes Received:
    5
    Try to get private proxies instead of public ones, you never know how many people are doing the same at a time with those proxies. Get above 50 to scrap 1000 urls.
     
  16. MissioneR

    MissioneR Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 8, 2012
    Messages:
    1,564
    Likes Received:
    156
    Location:
    In my GUTS
    I am using the proxies sold in Proxies for sale section which provide unlimited public proxies with prior test. I can able to scrape 1000s of proxies. Try it out.

    Lets await for GURUs reply. They can tell us the best way.
     
  17. blackhatninza

    blackhatninza Registered Member

    Joined:
    Jul 3, 2013
    Messages:
    97
    Likes Received:
    16
    I usually scrape around 50-60 public proxies when i need to do something on SB, Although private are always better.
     
  18. ghostbot

    ghostbot Registered Member

    Joined:
    Jul 3, 2013
    Messages:
    80
    Likes Received:
    5
    Depends actually, Why not use always private proxies. With that you dont have to bother everytime scrapping plus better success.