how to remove urls dont have .aspx type??scrapebpx

B

bleach

Senior Member

Oct 12, 2008

991

87

• Jun 10, 2010

• #1

i want to ask.
For example i scraped 10k of urls blog engine and i want to remove urls that dont have .aspx type
What can i do?

 

shylesson

Power Member

Jan 10, 2008

665

2,128

• Jun 10, 2010

• #2

Code:

http://textmechanic.com/Remove-Lines-of-Text-Containing.html

in the 'search for text' line, input the .aspx or w/e you want to contain in the urls, paste the list in the box, hit 'remove lines NOT containing' and boom--good to go.

 

B

BlackOut

Registered Member

Jan 25, 2008

55

31

• Jun 10, 2010

• #3

You can also use grep. Save the results to file and then grep .aspx file.txt > newfile.txt in Linux or use grep for windows.

 

B

BlackOut

Registered Member

Jan 25, 2008

55

31

• Jun 10, 2010

• #4

You can also use grep. Save the results to file and then grep .aspx file.txt > newfile.txt in Linux or use grep for Windows.

 

B

BlackOut

Registered Member

Jan 25, 2008

55

31

• Jun 10, 2010

• #5

You can also use grep. Save the results to file and then grep .aspx file.txt > newfile.txt in Linux or use grep for windows.

 

houseaz

Regular Member

Mar 25, 2010

312

80

• Jun 10, 2010

• #6

Does he have to use Grep 3 times though? Or is once enough?

 

B

BlackOut

Registered Member

Jan 25, 2008

55

31

• Jun 11, 2010

• #7

sorry...something went wrong while posting yesterday. maybe someone can delete these other posts.