Url Scraping of a website???

JohnKarr

Junior Member

Aug 12, 2009

127

22

• Sep 12, 2009

• #1

Hey guys,

I know how to get URL's based on keywords, with laserURL. But is there any program that will scape all the page url's of a specific website?

For ex.

lets say I have a digital camera offer, and I wanted every single URL that bestbuy.com has about digital cameras.

IS this possible, or is it something I need to get made? Any and all help is most appreciated, thanks.

 

Z

zorphee

Registered Member

May 16, 2009

71

7

• Sep 12, 2009

• #2

how does laserurl work? if it scrapes google results.. You can just type

site:bestbuy.com digital cameras

Click to expand...

anyway, would be great if you can share the laserurl software. I'm using something else that only scrapes yahoo..

 

Z

zorphee

Registered Member

May 16, 2009

71

7

• Sep 12, 2009

• #3

how does laserurl work? if it scrapes google results.. You can just type

site:bestbuy.com digital cameras

Click to expand...

anyway, would be great if you can share the laserurl software. I'm using something else that only scrapes yahoo..

 

H

hivo22700

Newbie

Sep 22, 2008

14

1

• Sep 12, 2009

• #4

There are a couple of http://www.blackhatworld.com/blackhat-seo/member-downloads/110168-get-scraper-pack.html

I'm new to coding and scraping. I would love to know how to scrape the URLs for an entire site and export them to a csv file.

I basically want to scrape all the external links from one particular site to see where they are linking to. Does anybody know of any tools that could help me do this?

 

R

revelation

Newbie

May 22, 2009

29

601

• Sep 12, 2009

• #5

There's a Firefox add-on called Link Gopher I believe. Might be what you're looking for.

 

nambooooo

Regular Member

May 21, 2009

215

72

• Sep 12, 2009

• #6

Try Web Data Extractor, you can find a cracked version around. It's pretty nice, you give the start URL and select what you wanna scrape off that site like URLs, emails, telephone nrs. Just play with the settings a little bit, but it's pretty intuitive.

 

JohnKarr

Junior Member

Aug 12, 2009

127

22

• Sep 12, 2009
• Thread Starter Thread Starter

• #7

It's in the forum just search for it

zorphee said:

how does laserurl work? if it scrapes google results.. You can just type


anyway, would be great if you can share the laserurl software. I'm using something else that only scrapes yahoo..

Click to expand...

 

H

hivo22700

Newbie

Sep 22, 2008

14

1

• Sep 12, 2009

• #8

@nambooooo - thanks that looks like exactly the type of tool I was looking for. I've had a look through the documentation, do you know if there is a way I can filter out "no follow" links from the results?

@revelation, thanks - looks like a pretty good tool to quickly extract data.

I think the Laser URL software is free anyway. http://www.mattharward.com/ who made it, he's got some other good tools on his site.

 

nambooooo

Regular Member

May 21, 2009

215

72

• Sep 12, 2009

• #9

@hivo22700 no, Web Data Extractor is not gonna help you on that, it's just a scraper, not a SEO tool.

 

T

techsmart

Newbie

Mar 3, 2009

20

0

• Jun 22, 2010

• #10

Last time I was actually looking for an automated scraping tool on google. I found Automation Anywhere, it has pretty nice capability to scrape data from websites. I have used it to scrape html tables, contents & even URL (as your case). You might want to try this http://www.automationanywhere.com/solutions/screenScrape.htm.

 

Last edited: Jun 22, 2010

risefromdeath

BANNED

Jul 1, 2009

651

107

• Jun 22, 2010

• #11

isnt it the wrong place for this thread?