hrefer scraping

[cdc515]

does anyone know how long it normally take to scrape a keyword list of 79168 words with 127 additive words? i know its a lot but i figured what they hell make a huge list. I know it matters on pc and speed as well so i am running it on verizon business DSL line and my pc it very fast.

Just some estimates of what you think would be nice.

Thanks

 

[dzoniij]

I harvested last night with 58 additive words and 85K word list took 11 hours to complete.

 

[cdc515]

what type of connection are you running and how many threads did you use?

 

[simplyrahul]

Yes I would like to know that as well.

 

[dzoniij]

what type of connection are you running and how many threads did you use?

200mbit/s fiber optic connection without proxy changing IP automatically every 10 min. Max 500 threads.

 

[Xrumer]

200mbit/s fiber optic connection without proxy changing IP automatically every 10 min. Max 500 threads.

Hey dzoniij.

The speed of your connection is super fast.

Have a curious question:
How many links can collect in your href month if you work 24/7?

 

[dzoniij]

Hey dzoniij.

The speed of your connection is super fast.

Have a curious question:
How many links can collect in your href month if you work 24/7?

I don't leave hrefer running 24/7 very long time. It is not necessary also. Basically I load 250K wordlist ( in this list are also most popular 40K english words) leave it at night and in morning I have new list. If I do again harvesting same day then I can't harvest almost no new forums. If I repeat process after 1-2 weeks then again in google index are new forums and I can get some new links. In short no point running hrefer 24/7 month because simply I don't have enough words and footprints for hrefer.