1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

My tough days for proxy and scraping, my way to challenge big G

Discussion in 'Black Hat SEO' started by wade82, Apr 1, 2014.

  1. wade82

    wade82 Registered Member

    Joined:
    Oct 4, 2013
    Messages:
    77
    Likes Received:
    10
    Recently I plan to scrape 100 Million links per day to do some spam project. Yes, after f*cked by big G for many years, I just wonder why i can't f*uck back.
    So I just do it.
    I purchased 2 GSA 2 XR and 2 VPS plus 1 dedicated server. Yes, big G is tough, so at least i want to arm myself first.
    But now I am stuck by the proxy. I tried to search the whole BHW but don't find any better solution, so I made a post here for help.
    Sure, I offer my tries to save some guys time. I will list what I tried.
    1) Public proxies list offered by BHW.
    1.a) The famous guy is proxygo, but the guy refuse to sell his service to low post members. Ok, for non enligh native speakers, post is kinda hard, so i can't count on him even i want to donate the BHW to show I am not a scamer.
    1.b) However proxygo do post many tips for port scan of public proxies. That is really great and nice. However I tried for a whole night and give up to scan by myself. Why? proxygo share the great tips of which software and some tips of scan, but you still don't know where to scan. And it is very rude to ask for the IP range, since that is his business. So the way is end here.
    1.c Some other guy here like TheBigProxyList.com is also offer great service. However their list is kinda few, for my goal 10 million per day, the list is not good enough.
    2) Public proxies via hrefer.
    Hrefer can scape proxy by itself, it is free and work great for small task, like 0.5 million per day. But for my goal 100 million, it doesn't help.
    3) revised proxy
    3.a There are 2 products here. one is reverseproxies.com. I tired the guy's product, his product work perfect for submitting like GSA, Xrumer etc since his proxy is really fast. Almost as fast as private proxy. but for pass google for my large goal, it doesn't help. Big G is hard to challenge. Also the custom service is really really really good on reverseproxies.com. Be aware of the 3 really for a non english native speaker, that is a highest review, since i don't know the other better words, lol.
    3.b proxyrack.com Now it is my solution here, it helps me scrape 3 million links per day, the best I have now. I am asking for the guy to offer a custom plan for me, since the existing plan don't meet my requirement, I need larger. But now they are upgrading their system, so i need to wait for 2-3 days.
    4) private or semi private proxy. They are expensive, but if this is the last choice, I have to step on.
    4a. semi private proxy, i purchased semi private proxy (shared plan via buyproxies.org) it can scrape about 0.3M per day. Yea, good for small project not for me.
    4b. private proxy.
    I search the sale thread and list targeted proxy below.

    PP-100 | $169.00 / Monthly
    www.proxy-n-vpn.com

    SPP-100 | $164.00 / Monthly
    www.sslprivateproxy.com

    100 HTTP proxies $150
    buyproxies.org

    100 Private Proxies - $120/m
    Exclusive BHW Discount:
    * Order with the promo code "bhwonly" to receive an exclusive 25% discount on private proxies.
    SquidProxies.com

    60 Semi Private Proxiescheapprivateproxies.org
    http://cheapprivateproxies.org
    $42.00 USD
    Use this coupon code "CPP-BHW-12OFF" & get 12% OFF Life Time Discount on all Packages Specially for BHW Forum Members.

    I just purchased 60 Semi Private Proxies cheapprivateproxies.org and 100 Private Proxies - $120/m on SquidProxies.com.
    Yes, the cheapest ones I can find here. I still haven't got them, but i will make a post when i get and test them.

    For my project, actually i don't think Private Proxies are good ideas. But now it seems i have no other solution.
    Any one here can help? thanks.
     
  2. StraussCan

    StraussCan Regular Member

    Joined:
    Aug 24, 2013
    Messages:
    393
    Likes Received:
    128
    Occupation:
    Pen testing
    seems like self promotion
     
  3. wade82

    wade82 Registered Member

    Joined:
    Oct 4, 2013
    Messages:
    77
    Likes Received:
    10
    Just for fun, guy.
    Ok, result come out.
    For 60 Semi Private Proxiescheapprivateproxies.org
    http://cheapprivateproxies.org
    $42.00 USD
    It is semi private, so all the proxies are not fit my project. I contact the customer service and get refund really fast.

    For SquidProxies.com, i got 100 proviate proxies, the first SB test, I only get 28 google passed proxies, i request to change the others. And still haven't reply.
    It seems private proxies are not a good idea.
    Anyone help here?
     
  4. wade82

    wade82 Registered Member

    Joined:
    Oct 4, 2013
    Messages:
    77
    Likes Received:
    10
    Update:
    I ran the 100 proxies for a few minutes, in order to avoid burning the proxies, i set the threads to be 150. After around 30 minutes run.
    I tested it again, the google passed proxies are around 18-30.
    For 30 minutes, I only scrape 20000 links. I need other solution.
     
  5. wade82

    wade82 Registered Member

    Joined:
    Oct 4, 2013
    Messages:
    77
    Likes Received:
    10
    update:
    Ok, now 1 hours passed. with 28 google passed private proxies, I scrape 30000 links. The result is unacceptable for me.
    Also I tested the 100 proxies the last time, the result get me confused, now it is 45 google passed proxies. It seems the proxies recover now.
    but I don't plan to go on with these proxies, since the speed don't meet my goal.
    Help, anyone know how to work out the problem? Thanks.
     
    Last edited: Apr 1, 2014
  6. wade82

    wade82 Registered Member

    Joined:
    Oct 4, 2013
    Messages:
    77
    Likes Received:
    10
    Update:
    Ok, it is kinda crazy to post so many update here in one day.
    but I got reply from SquidProxies.com, they changed a new google passed set proxies for me.
    I tested them again before i run the proxies, now i have 87 google passed proxies, so I ran hrefer again, now the scrape speed is about 1500-2000 per minute.
    That is bottom line I can accept, so I am going to keep them.
    but I request to change all the RO proxies to US, I think that would increase the speed further.
    My only concern is these proxies can't last long, so I set fewer thread to avoid burning my proxies. Hope it works.

    Let me run the hrefer for 24 hours for a more stable test. I will update tomorrow.
    If these work. I can get about 2 million links per day for each instance of hrefer. Looks like I still have a long way to go.
     
    Last edited: Apr 1, 2014
  7. wade82

    wade82 Registered Member

    Joined:
    Oct 4, 2013
    Messages:
    77
    Likes Received:
    10
    look like no one interested in my post. lol, so this is my last update and share.
    For private and pulic, now conclusion is pulic. But you need thousands of proxies.
    Finally I can finish my project with the help of my friend. 100 Million is huge, but it still doable. Just cost is kinda huge.