Looking for a program to clean urls

[ProMonkey]

Hey,

Im currently scraping URLs from google and bing but i need to clean the list so the domains are

Unique (no duplicates in the list)
remove everything after the .com/page (as we only want the index)

Does anyone know of a program that will do this? The list will be big (millions of urls)

 

[Gogol]

I bet Notepad++ could do that for you, and it does support large files.


As for removing dups, go to Replace (CTRL + H) in notepad++, select search mode as Regular expression and in search string, place:

^(.*?)$\s+?^(?=.*^\1$)

Keep the replace textbox empty. Click on Replace All.


As for keeping the domain from the URL, go to Replace (CTRL + H) in notepad++, select search mode as Regular expression and in search string, place:

^(?:https?://)?([^/.]+(?=\.)|)(\.?[^/.]+\.[^/]+)/?(.+|)$

In replace textbox place

\1 \2

Click on Replace All.


Hope it helps

 

[SiddharthW]

scarpebox do it easily

 

[ProMonkey]

Cheers im a novice but ill give it a go thanks

 

[Gogol]

scarpebox do it easily

Yeah but it's not free although it is worth every penny spent, especially with the exclusive bhw discount. Notepad++ is pretty powerful when it comes to Search & Replace and most importantly, it is free

 

[ProMonkey]

yeh just checking out some videos on scarpebox i think that could do the job

 

[HoNeYBiRD]

Dupe Remove: http://www.scrapebox.com/free-dupe-remove. Trimming to root domain can be done in Excel by splitting lines using the "/" delimiter or in Notepad++ using regex, which was explained above.

 

[tompots]

As stated by @Gogol you can use a regex, most people in IM would use a tool such a scrapeboxhttps://www.blackhatworld.com/members/gogol.137386/

 

[ScrapingProxies]

Scrapebox all the way - it can handle extremely large url list, it has function that allows you to trim the URL to a certain degree, and it can also dedupe (remove duplicates) from the list as well.
The update that they did not long ago, enable their software to process millions of URLs - or you can also use a sub-plugin that is free, that has higher reliability when processing extremely large lists.

 

[DunDidIt2X]

Does anyone know of a program that will do this? The list will be big (millions of urls)

Some of that will also depend on your machine specs as far as processing millions of url's. You would prolly do better splitting the list up in smaller sets, especially if you are using a machine with very little processing power.

 

[ProMonkey]

yeh looking like SB might be the way forward, is it possible to upload a url list i already have? Yes i could split the links up.