Remove dup lines from nodes.txt

[BlogPro]

http://www.heypasteit.com/clip/32R7
For example , in this link , there are 2 duplicate lines (by lines i mean , url and after url the content) , if it has same content and diffrent url is not duplicate ..
Windows 7 / Kali linux

The problem is, you require relative de-duplication.

Any command such as the one @Pakal posted above or sort or even uniq will run a de-duplication where every line is considered a single node.

So if it has the same content, irrespective of it being different URLs, it will be removed.

You'll need regex and a much more complicated system to run relative de-duplication.

Or you'll need to run a work around and first bring them into a single line with a seperator.

So

http://ideografhmata.gr/forum/viewtopic.php?f=142&t=17507
Πολιτική-Κοινωνικά Κινήματα<||>Για τις συνάξεις του ΠΑΣΟΚ ελεω επετείου

becomes

http://ideografhmata.gr/forum/viewtopic.php?f=142&t=17507 > Πολιτική-Κοινωνικά Κινήματα<||>Για τις συνάξεις του ΠΑΣΟΚ ελεω επετείου

Then run either sort or uniq and then replace the seperator with a carriage return.

 

[MariosElGreco]

The problem is, you require relative de-duplication.

Any command such as the one @Pakal posted above or sort or even uniq will run a de-duplication where every line is considered a single node.

So if it has the same content, irrespective of it being different URLs, it will be removed.

You'll need regex and a much more complicated system to run relative de-duplication.

Or you'll need to run a work around and first bring them into a single line with a seperator.

So

http://ideografhmata.gr/forum/viewtopic.php?f=142&t=17507
Πολιτική-Κοινωνικά Κινήματα<||>Για τις συνάξεις του ΠΑΣΟΚ ελεω επετείου

becomes

http://ideografhmata.gr/forum/viewtopic.php?f=142&t=17507 > Πολιτική-Κοινωνικά Κινήματα<||>Για τις συνάξεις του ΠΑΣΟΚ ελεω επετείου

Then run either sort or uniq and then replace the seperator with a carriage return.

True -- and also this file is on multiple language , and after the de-dup i get some "@$#%$#^%$^^^" .. So yes , i still have problem , and now the xrumer cannot load the database . (i have backup ofcourse) , but it still doesen't work as we want with linux

 

[BlogPro]

It is complicated, not impossible.

Consider hiring a developer, they should be able to spin up some regex for you real quick that will do what you want done.