1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scrapebox Yahoo scraping sucks lately.... any suggestions?

Discussion in 'Black Hat SEO' started by malicious, Jul 31, 2011.

  1. malicious

    malicious Newbie

    Joined:
    Nov 9, 2008
    Messages:
    24
    Likes Received:
    8
    Hello people, the last days i get 502 errors while scraping big keyword lists on yahoo, even setting down the connections didnt help.
    So is the only workaround at the moment to get more proxies to get that error later?
    its everytime 50-70mb i can scrape then it stops and i have tried to set the connection settings to 5, but didnt see a difference. Did anyone else have better success with other settings or even not experience this problem??? Im on the lastest version...
    Edit: im starting to scrape some proxies with the new harvester with the checkbox to ignore google and will add the fastest ones to my list and see how much it iwll increase...
    EDIT 2: Solved it myself... haha:D
    So i did scrape some public proxies and added them to my 50 shared ones. All those harvested with 15s timeout and 30 connections. once it got to 250 i stopped testing to start the yahoo harvesting test. It was very slow..... 8 urls/s avg. before it was allready low at around 25 urls/s.
    Now i did change the yahoo harvester connections to 30 and voila, its scraping with 106 urls/s and has allready passed that limit i had before....
    So if anybody runs into the same problem just use these settings and enough fast proxies.
     
    Last edited: Jul 31, 2011
  2. Waxler

    Waxler Registered Member

    Joined:
    Jul 17, 2010
    Messages:
    52
    Likes Received:
    4
    Thanks for posting this, I got some public proxies and now it works prefectly.
     
  3. myfault

    myfault Power Member

    Joined:
    Sep 21, 2012
    Messages:
    636
    Likes Received:
    121
    old is gold. I think i will try this method to solve the problem.