Looking for a large number of urls

[ProMonkey]

Hi,

Im looking for a large list of URLs for a program i am developing (the bigger the list the better)
the links dont need to be any particular category.

Does anyone have a list knocking about they could send me?

Cheers

 

[proxified]

Hi,

Im looking for a large list of URLs for a program i am developing (the bigger the list the better)
the links dont need to be any particular category.

Does anyone have a list knocking about they could send me?

Cheers

You can always use a scraper.

 

[ProMonkey]

Yes currently have one running, thought it would be worth asking if someone had a pre scraped list as it will take about a month to scrape the total links i need

 

[Cleast Intwood]

Knowing id really appreciate this as a newcomer to this forum, sure i can help you out. How many links u need?

Edit: Lol i see uve been here wayy longer than me

 

[RightFootFanatic]

Fetch the sitemaps from multiple large stores or magazines

 

[ProMonkey]

Long time user first-time poster haha as many as you have or can get your hands on. We just need the index page of sites, not every page on there.

The tool we are creating is like a search engine but it's not a search engine

 

[Alex D.]

You can easily do that with Regex.

 

[ProMonkey]

Alex, could you advise how?

 

[Alex D.]

Alex, could you advise how?

I would suggest notepad++ but your file is very big. I wouldn't store that much data in txt file. Better use real database and write simple script to update urls. I would personally do that with C#, it's pretty much easy task.

 

[ProMonkey]

Would you be willing to wright the code for me alex? i think the database idea is better too.

 

[Alex D.]

Would you be willing to wright the code for me alex? i think the database idea is better too.

Contact me on Skype: alexscraping or here via PM

 

[ProMonkey]

sent you a PM

 

[HoNeYBiRD]

Have you checked the download section? People shared there quite big packages with millions of urls occasionally.

 

[ProMonkey]

Ive not HoNeYBiRD will take a look now thank you

 

[Cleast Intwood]

I reckon this should be enough urls if u use all these right?

https ://www.goog le.com/sitemap.xml

 

[ProMonkey]

I have looked in the download section, looks like all the lists dont have active download links shame as a big pre scraped list would be ideal

 

[bartosimpsonio]

Hi,

Im looking for a large list of URLs for a program i am developing (the bigger the list the better)
the links dont need to be any particular category.

Does anyone have a list knocking about they could send me?

Cheers

Go here

 

[ProMonkey]

Go here

Legend mate that will get me started well

 

[seotrick007]

Hi,

Im looking for a large list of URLs for a program i am developing (the bigger the list the better)
the links dont need to be any particular category.

Does anyone have a list knocking about they could send me?

Cheers

- I can do this for you, if you want. I'll be glad to help you.
But , I think the real section of this post will be: HIRE FREELANCER.

 

[ProMonkey]

- I can do this for you, if you want. I'll be glad to help you.
But , I think the real section of this post will be: HIRE FREELANCER.

I can scrape them myself, its just taking a long time. I was wondering if anyone had old lists, they wont be used for anything apart from collecting data.