URL List Editor Needed

[TragikGod]

If someone can help me with this I would be VERY appreciative.

I'm hoping someone can help me out with a program that is powerful enough to...
1. Load a URL list. (List 1)
2. Load a 1mill+ URL list. (List 2)
3. Remove (List 2) from (List 1)
4. Save (List 3) which is (List 1 - List 2) = List 3

OR

Something like Scrapebox's exclude list but on a bigger scale.

I looked all over the internet only to find editiors that just find the difference and highlight it and UNIX stuff.

Thank you in advance.

 

[RedLable]

Ubot can do this, might have to make the larger list smaller depending on the memory your computer has tho.

 

[fatboy]

What do you want it written in - Ubot in its default form will most definitely choke and die if you throw that many links at it at one go, perhaps with the decent plugins it may work.

If you want to run scripts on Linux, then Perl will do it in a heartbeat....

Give more details and you will probably get better answers

 

[Conor]

Use excel. Just paste all the URLs in one column, and then click "Remove Duplicates'. Then simply copy the remaining URLs into notepad and save it.

If excel can't handle that many URLs, try Notepad++ with the TextFX plugin: http://www.cathrinewilhelmsen.net/2...e-blank-lines-and-sort-data-in-one-operation/

 

[Wilson Grant Fisk]

Notepad++ with TextFX plugin

http://notepad-plus-plus.org/
http://sourceforge.net/projects/npp-plugins/files/TextFX/TextFX v0.26/



EDIT: Tree beat me to it

 

[WMAid]

I have one simple script which all my friends and me use very often.

Free. Open Source
bit.ly/1voiZU5

you need to install python 2.7 on your machine to run it.

Script works with 3 files

• i.txt - input data
• d.txt - delete from input data
• o.txt - output data

Also it can delete not unique lines.

 

[Hawk Eye]

I will help you,Check your skype.

 

[Wilson Grant Fisk]

I have one simple script which all my friends and me use very often.

Free. Open Source
bit.ly/1voiZU5

you need to install python 2.7 on your machine to run it.

Script works with 3 files

• i.txt - input data
• d.txt - delete from input data
• o.txt - output data

Also it can delete not unique lines.

It is against the rules to link directly to a download and hide the link with a URL shortener and you also need to provide a VirusTotal report

https://www.virustotal.com/en-gb/fi...1ce9e82fedb5a06ddae2ce76/analysis/1411812007/

 

[WMAid]

It is against the rules to link directly to a download and hide the link with a URL shortener and you also need to provide a VirusTotal report

Thanks for the virus total report.
I'm newbie here and still can't pos tnormal links. URL shortener helped me to help topic starter.
I don't mind if moderators will remove or change my link.

 

[Wilson Grant Fisk]

Thanks for the virus total report.
I'm newbie here and still can't pos tnormal links. URL shortener helped me to help topic starter.
I don't mind if moderators will remove or change my link.

No problem Mate, thanks for sharing but just familiarise yourself with the Download rules
http://www.blackhatworld.com/blackhat-seo/f29-member-downloads/

 

[gary2]

even i will suggest Ubot like redlable... dependable and good service

 

[misteryou.]

maybe i have not understand correctly but
http://textmechanic.com

 

[Xtremerank]

Use excel. Just paste all the URLs in one column, and then click "Remove Duplicates'. Then simply copy the remaining URLs into notepad and save it.

If excel can't handle that many URLs, try Notepad++ with the TextFX plugin: http://www.cathrinewilhelmsen.net/2012/05/16/notepad-remove-duplicates-remove-blank-lines-and-sort-data-in-one-operation/

Excel is limited to 1m rows unfortunately
TextFX is awesome though, recommended!