hrefer scraping

Discussion in 'Black Hat SEO Tools' started by cdc515, Mar 3, 2010.

  1. cdc515

    cdc515 Power Member

    Joined:
    Nov 23, 2009
    Messages:
    602
    Likes Received:
    93
    does anyone know how long it normally take to scrape a keyword list of 79168 words with 127 additive words? i know its a lot but i figured what they hell make a huge list. I know it matters on pc and speed as well so i am running it on verizon business DSL line and my pc it very fast.

    Just some estimates of what you think would be nice.

    Thanks
     
  2. dzoniij

    dzoniij Regular Member

    Joined:
    May 24, 2009
    Messages:
    469
    Likes Received:
    111
    I harvested last night with 58 additive words and 85K word list took 11 hours to complete.
     
  3. cdc515

    cdc515 Power Member

    Joined:
    Nov 23, 2009
    Messages:
    602
    Likes Received:
    93
    what type of connection are you running and how many threads did you use?
     
  4. simplyrahul

    simplyrahul Registered Member

    Joined:
    Jun 17, 2008
    Messages:
    90
    Likes Received:
    15
    Yes I would like to know that as well.
     
  5. dzoniij

    dzoniij Regular Member

    Joined:
    May 24, 2009
    Messages:
    469
    Likes Received:
    111
    200mbit/s fiber optic connection without proxy changing IP automatically every 10 min. Max 500 threads.
     
  6. Xrumer

    Xrumer Junior Member

    Joined:
    Jan 13, 2010
    Messages:
    141
    Likes Received:
    53
    Hey dzoniij.

    The speed of your connection is super fast. :)

    Have a curious question:
    How many links can collect in your href month if you work 24/7?
     
  7. dzoniij

    dzoniij Regular Member

    Joined:
    May 24, 2009
    Messages:
    469
    Likes Received:
    111
    I don't leave hrefer running 24/7 very long time. It is not necessary also. Basically I load 250K wordlist ( in this list are also most popular 40K english words) leave it at night and in morning I have new list. If I do again harvesting same day then I can't harvest almost no new forums. If I repeat process after 1-2 weeks then again in google index are new forums and I can get some new links. In short no point running hrefer 24/7 month because simply I don't have enough words and footprints for hrefer.