Remove duplicates in 2 variables?

J

Jimm22

Registered Member

Jul 15, 2021

73

51

• Jul 30, 2021

• #1

Hello,
I have list of spammed urls on this format:

Anchor|Urls

What I wish to do here is to remove from the list every "unique urls" (Urls that appears only once) and only keep lines which have duplicated URLs (URLs appearing multiple times, maybe even under different anchor).

Do you know if there is a tool or something to do this?

 

tazarbm

Elite Member

Executive VIP

Jr. VIP

Oct 28, 2020

11,663

14,618

• Jul 30, 2021

• #2

you can do this with excel, just don't ask me how cause I don't know. All I know is that I had to do a similar thing 5-6 years ago and I went to the mr. excel forum and asked for guidance and they've told me how to do it. It's pretty easy if you have basic excel knowledge, but my knowledge is not even basic

 

uncutu

Elite Member

Aug 6, 2010

3,506

3,374

• Jul 30, 2021

• #3

https://www.google.com/search?q=list+keep+only+duplicate+lines
pick ur tutorial. you'll need ms excel.

 

J

Jimm22

Registered Member

Jul 15, 2021

73

51

• Jul 30, 2021
• Thread Starter Thread Starter

• #4

uncutu said:

https://www.google.com/search?q=list+keep+only+duplicate+lines
pick ur tutorial. you'll need ms excel.

Click to expand...

The problem with this tutorial is that only the url is removed, while I need anchor + url to be removed

 

mdevo

Banned - promoting another forum via PM

Jul 10, 2020

126

156

• Jul 30, 2021

• #5

I am pretty sure it is doable with EmEditor. However, I don't think that you will find any tutorial how to do it. I'd do it like:

1. Divide it into columns by selecting delimiter as "|"
2. In second column bookmark all duplicate strings (URLs)
3. Inverse bookmarked lines, so now you will have bookmarked only unique URLs
4. Delete bookmarked lines
5. Done!

 

uncutu

Elite Member

Aug 6, 2010

3,506

3,374

• Jul 30, 2021

• #6

Jimm22 said:

The problem with this tutorial is that only the url is removed, while I need anchor + url to be removed

Click to expand...

are they separated by a pipe, comma, or what? merge it all into one line (or row)
run the tutorial
it should do its magic irrelevant if its a url or plain text
when its done, set your delimiter in excel to split the anchor and url back into two rows
or keep trying other tutorials. you'll eventually get it

 

Lightworkerman

Newbie

Feb 29, 2020

45

7

• Aug 4, 2021

• #7

I know plenty of ways of how to do that in python. I didn't know that even excel is capable of it. Why don't you just google for the solution of your problem?