Something to extract URL's from a .txt file

K

Kronz

Newbie

Oct 1, 2014

10

1

• Nov 20, 2014

• #1

Basically I have a large .txt file that contains various URL's mixed with some text.

I want to extract those URL's.

I googled hard, but I couldn't find a free program that does that simple task.

Please help.

 

mktanny

Regular Member

Oct 22, 2009

226

62

• Nov 20, 2014

• #2

Try this (you will need notepad ++) -- http://stackoverflow.com/a/8477855/3117013

 

S

scriptdaddy

Newbie

Nov 17, 2014

1

0

• Nov 20, 2014

• #3

You can use a regex for that easily, probably in python.

 

todordonev

Power Member

Nov 23, 2012

520

485

• Nov 20, 2014

• #4

Notepad++ and use regex in the find window (ctrl+f)
You can also google textmechanic and do the following:
Make new line before http
Replace empty space with new line (cause the urls are separated with empty space otherwise they will merge and wont work)
Remove lines that dont contain http
Remove duplicate lines.
Save to txt.

 

Davidnrog

Regular Member

Apr 21, 2014

218

32

• Nov 20, 2014

• #5

I'll help you, Just pin me your .txt file..

 

ItsBlinkHere

Regular Member

Apr 27, 2014

408

150

• Nov 20, 2014

• #6

Kronz said:

Basically I have a large .txt file that contains various URL's mixed with some text.

I want to extract those URL's.

I googled hard, but I couldn't find a free program that does that simple task.

Please help.

Click to expand...

Post a sample...it shouldn't be to hard to code up.

 

T

tornadododo

Newbie

Nov 20, 2014

22

4

• Nov 20, 2014

• #7

get application from the apple store

 

S

seohosting

Newbie

Nov 19, 2014

23

8

• Nov 20, 2014

• #8

If you are using google chrome you can download the "Jon's all in one domain tool" extension. Has a great simple domain cleaner built in.

You can also find a domain cleaner over at function.me

 

hpasha

Elite Member

Jr. VIP

May 15, 2011

1,505

203

• Nov 20, 2014

• #9

put that txt file into scrapebox. it will extract the urls and then check their headers

 

Wilson Grant Fisk

Elite Member

Nov 10, 2012

15,677

46,847

• Nov 20, 2014

• #10

try this: http://free.groupfetch.com/products/ugrab/index.php

 

Aty

Elite Member

Jan 27, 2011

7,631

5,301

• Nov 21, 2014

• #11

or this: http://url-extractor.software.informer.com/download/

 

K

Kronz

Newbie

Oct 1, 2014

10

1

• Nov 21, 2014
• Thread Starter Thread Starter

• #12

W130SN said:

try this:

Click to expand...

Thank you very much. It saved me.

Only problem is it is a bit slow. I have been trying to process a 200MB file today and it hasn't even extracted 1/8 of the links in 5 hours.

or this:

Click to expand...

Thanks, but this one crashes.

 

A

Atomic76

Registered Member

May 24, 2014

65

39

• Nov 21, 2014

• #13

Is the text file in some sort of consistent format? Such as a comma separated or tab separated? If so, you can just paste it into Excel and do a Data > Text To Columns, and select what the delimiter is.

 

prab1996

Elite Member

Jan 8, 2013

3,492

2,044

• Nov 21, 2014

• #14

pm i might do it for free if it will not consume much time. i do this type of work every day.
-=-

 

C

COLDEXE

Junior Member

Aug 29, 2013

104

24

• Nov 21, 2014

• #15

I'll do this for free if you like, will only take a few minutes as I've written similar software. I've sent you a pm anyway mate.

 

botrockets

Regular Member

Mar 16, 2013

389

647

• Nov 23, 2014

• #16

I have created very fast tool for such purpose. You can extract links from text file of any size.. may it be hundreds mbs, Gb or TB

here's screenshot :

See this post: http://www.blackhatworld.com/blackh...ur-reasonable-ideas-tools-15.html#post7408391