need tool for filter huge urls text file

Botsky said:

Write to me

Click to expand...

fuad_2000 said:

what mean write to me ????????

Click to expand...

fuad_2000 said:

hi guys

I have a text file containing 100.000.000 url's and i want filter this file
i have used scrapebox for filter it but scrapebox not support biggest file

i need help for find tool can support huge file for i can filter it from Duplicate url's

thank you

Click to expand...

loopline said:

Just use the scrapebox dupe remove addon, and split the file into like 50 million or 25 million urls. Then filter it, then you can use the same addon to put the filter back together (if you want it back together). The addon is supposed to support upto like 180 million lines.

Click to expand...

fuad_2000 said:

hi loopline
i find it and i will test it and feedback you

Click to expand...

bossofthebosses said:

How do u want to filter the file? Like deleting duplicate lines? Any other operation u want to do with it?

Click to expand...

fuad_2000 said:

deleting duplicate lines

Click to expand...