Looking for a program to clean urls

ProMonkey

Junior Member

Jun 8, 2012

100

9

• May 30, 2018

• #1

Hey,

Im currently scraping URLs from google and bing but i need to clean the list so the domains are

Unique (no duplicates in the list)
remove everything after the .com/page (as we only want the index)

Does anyone know of a program that will do this? The list will be big (millions of urls)

 

Gogol

Administrator

Staff member

Moderator

Executive VIP

Jr. VIP

Sep 10, 2010

11,807

26,658

• May 30, 2018

• #2

I bet Notepad++ could do that for you, and it does support large files.


As for removing dups, go to Replace (CTRL + H) in notepad++, select search mode as Regular expression and in search string, place:

Code:

^(.*?)$\s+?^(?=.*^\1$)

Keep the replace textbox empty. Click on Replace All.


As for keeping the domain from the URL, go to Replace (CTRL + H) in notepad++, select search mode as Regular expression and in search string, place:

Code:

^(?:https?://)?([^/.]+(?=\.)|)(\.?[^/.]+\.[^/]+)/?(.+|)$

In replace textbox place

Code:

\1 \2

Click on Replace All.


Hope it helps

 

Last edited: May 30, 2018

SiddharthW

Power Member

Jul 29, 2017

734

384

• May 30, 2018

• #3

scarpebox do it easily

 

ProMonkey

Junior Member

Jun 8, 2012

100

9

• May 30, 2018
• Thread Starter Thread Starter

• #4

Cheers im a novice but ill give it a go thanks

 

Gogol

Administrator

Staff member

Moderator

Executive VIP

Jr. VIP

Sep 10, 2010

11,807

26,658

• May 30, 2018

• #5

SiddharthW said:

scarpebox do it easily

Click to expand...

Yeah but it's not free although it is worth every penny spent, especially with the exclusive bhw discount. Notepad++ is pretty powerful when it comes to Search & Replace and most importantly, it is free

 

Last edited: May 30, 2018

ProMonkey

Junior Member

Jun 8, 2012

100

9

• May 30, 2018
• Thread Starter Thread Starter

• #6

yeh just checking out some videos on scarpebox i think that could do the job

 

HoNeYBiRD

Elite Member

May 1, 2009

10,187

12,580

• May 30, 2018

• #7

Dupe Remove: http://www.scrapebox.com/free-dupe-remove. Trimming to root domain can be done in Excel by splitting lines using the "/" delimiter or in Notepad++ using regex, which was explained above.

 

tompots

Elite Member

Dec 11, 2011

5,775

4,871

• May 30, 2018

• #8

As stated by @Gogol you can use a regex, most people in IM would use a tool such a scrapeboxhttps://www.blackhatworld.com/members/gogol.137386/

 

ScrapingProxies

Regular Member

Apr 30, 2018

367

27

• May 31, 2018

• #9

Scrapebox all the way - it can handle extremely large url list, it has function that allows you to trim the URL to a certain degree, and it can also dedupe (remove duplicates) from the list as well.
The update that they did not long ago, enable their software to process millions of URLs - or you can also use a sub-plugin that is free, that has higher reliability when processing extremely large lists.

 

DunDidIt2X

Elite Member

Mar 22, 2012

2,863

2,432

• May 31, 2018

• #10

ProMonkey said:

Does anyone know of a program that will do this? The list will be big (millions of urls)

Click to expand...

Some of that will also depend on your machine specs as far as processing millions of url's. You would prolly do better splitting the list up in smaller sets, especially if you are using a machine with very little processing power.

 

ProMonkey

Junior Member

Jun 8, 2012

100

9

• May 31, 2018
• Thread Starter Thread Starter

• #11

yeh looking like SB might be the way forward, is it possible to upload a url list i already have? Yes i could split the links up.