1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

gscraper slow

Discussion in 'Black Hat SEO' started by cozziola, Jun 29, 2014.

  1. cozziola

    cozziola Regular Member

    Joined:
    May 20, 2014
    Messages:
    365
    Likes Received:
    18
    My average speed for gscraper is only 2400 / minute. I have a pretty fast vps. I am using gscraper server proxies. I was using my own 50 private ones until i found out that this was a terrible idea.

    I really need to speed this up because i am only 0.28% through this intended scrape after 50 hours.

    Hopefully someone can help me crack this mofo :)
     
  2. myfault

    myfault Power Member

    Joined:
    Sep 21, 2012
    Messages:
    636
    Likes Received:
    121
    Gscraper server proxies now a days were slow. You should try another service which is provided in bulk like Red-Proxy and Goproxy. What is your vps spec ?
     
  3. Zombie Pop

    Zombie Pop Regular Member

    Joined:
    Dec 18, 2013
    Messages:
    360
    Likes Received:
    123
    you could also try reducing the total number of results to scrape for. if you have it set to 100 results per page and 1000 total results per query, depending on the amount of footprints and KW's you are using that could be A LOT of results you're telling gscraper to get. I don't know what your settings are, but for heavy scraping, you need more proxies. I've got 60 demi deditcated with 5 threads and get about the same amount as you said p/s with it set to get a lot of results, which wind up being mostly duplicate domains anyway.
     
  4. twilightofidols

    twilightofidols Power Member

    Joined:
    Nov 11, 2011
    Messages:
    675
    Likes Received:
    224
    It could be your footprints. I'm getting 50,000 URLs per minute right now with 1,000 threads on a dedi.
     
  5. Zombie Pop

    Zombie Pop Regular Member

    Joined:
    Dec 18, 2013
    Messages:
    360
    Likes Received:
    123
    ^ that too. gscraper highlights footprints in red that will be slow.
     
  6. cozziola

    cozziola Regular Member

    Joined:
    May 20, 2014
    Messages:
    365
    Likes Received:
    18
  7. Rambo the Great

    Rambo the Great Junior Member

    Joined:
    Jun 30, 2014
    Messages:
    154
    Likes Received:
    11
    hrefer ftw baby! :p
     
  8. cozziola

    cozziola Regular Member

    Joined:
    May 20, 2014
    Messages:
    365
    Likes Received:
    18
    Right i think my ave speed comes down as proxies die?

    Is there any way i can clean the proxy list as it is scraping?

    What is the best way to clean proxy lists?