1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

hrefer scraping

Discussion in 'Black Hat SEO Tools' started by cdc515, Mar 3, 2010.

  1. cdc515

    cdc515 Power Member

    Joined:
    Nov 23, 2009
    Messages:
    602
    Likes Received:
    92
    does anyone know how long it normally take to scrape a keyword list of 79168 words with 127 additive words? i know its a lot but i figured what they hell make a huge list. I know it matters on pc and speed as well so i am running it on verizon business DSL line and my pc it very fast.

    Just some estimates of what you think would be nice.

    Thanks
     
  2. dzoniij

    dzoniij Regular Member

    Joined:
    May 24, 2009
    Messages:
    469
    Likes Received:
    111
    I harvested last night with 58 additive words and 85K word list took 11 hours to complete.
     
  3. cdc515

    cdc515 Power Member

    Joined:
    Nov 23, 2009
    Messages:
    602
    Likes Received:
    92
    what type of connection are you running and how many threads did you use?
     
  4. simplyrahul

    simplyrahul Registered Member

    Joined:
    Jun 17, 2008
    Messages:
    90
    Likes Received:
    15
    Yes I would like to know that as well.
     
  5. dzoniij

    dzoniij Regular Member

    Joined:
    May 24, 2009
    Messages:
    469
    Likes Received:
    111
    200mbit/s fiber optic connection without proxy changing IP automatically every 10 min. Max 500 threads.
     
  6. Xrumer

    Xrumer Junior Member

    Joined:
    Jan 13, 2010
    Messages:
    140
    Likes Received:
    52
    Hey dzoniij.

    The speed of your connection is super fast. :)

    Have a curious question:
    How many links can collect in your href month if you work 24/7?
     
  7. dzoniij

    dzoniij Regular Member

    Joined:
    May 24, 2009
    Messages:
    469
    Likes Received:
    111
    I don't leave hrefer running 24/7 very long time. It is not necessary also. Basically I load 250K wordlist ( in this list are also most popular 40K english words) leave it at night and in morning I have new list. If I do again harvesting same day then I can't harvest almost no new forums. If I repeat process after 1-2 weeks then again in google index are new forums and I can get some new links. In short no point running hrefer 24/7 month because simply I don't have enough words and footprints for hrefer.