Looking for a large number of urls

ProMonkey

ProMonkey

Junior Member
Joined
Jun 8, 2012
Messages
100
Reaction score
9
Hi,

Im looking for a large list of URLs for a program i am developing (the bigger the list the better)
the links dont need to be any particular category.

Does anyone have a list knocking about they could send me? :)

Cheers
 
Advertise on BHW
ProMonkey said:
Hi,

Im looking for a large list of URLs for a program i am developing (the bigger the list the better)
the links dont need to be any particular category.

Does anyone have a list knocking about they could send me? :)

Cheers
Click to expand...

You can always use a scraper.
 
Yes currently have one running, thought it would be worth asking if someone had a pre scraped list as it will take about a month to scrape the total links i need :)
 
Knowing id really appreciate this as a newcomer to this forum, sure i can help you out. How many links u need?

Edit: Lol i see uve been here wayy longer than me :p
 
Long time user first-time poster haha :D as many as you have or can get your hands on. We just need the index page of sites, not every page on there.

The tool we are creating is like a search engine but it's not a search engine :P
 
Alex, could you advise how? :)
 
ProMonkey said:
Alex, could you advise how? :)
Click to expand...
I would suggest notepad++ but your file is very big. I wouldn't store that much data in txt file. Better use real database and write simple script to update urls. I would personally do that with C#, it's pretty much easy task.
 
Would you be willing to wright the code for me alex? :) i think the database idea is better too.
 
Have you checked the download section? People shared there quite big packages with millions of urls occasionally.
 
I have looked in the download section, looks like all the lists dont have active download links :( shame as a big pre scraped list would be ideal
 
ProMonkey said:
Hi,

Im looking for a large list of URLs for a program i am developing (the bigger the list the better)
the links dont need to be any particular category.

Does anyone have a list knocking about they could send me? :)

Cheers
Click to expand...
- I can do this for you, if you want. I'll be glad to help you.
But , I think the real section of this post will be: HIRE FREELANCER.
 
seotrick007 said:
- I can do this for you, if you want. I'll be glad to help you.
But , I think the real section of this post will be: HIRE FREELANCER.
Click to expand...
I can scrape them myself, its just taking a long time. I was wondering if anyone had old lists, they wont be used for anything apart from collecting data.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top