1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Url Scraping of a website???

Discussion in 'Pay Per View' started by JohnKarr, Sep 12, 2009.

  1. JohnKarr

    JohnKarr Junior Member

    Joined:
    Aug 12, 2009
    Messages:
    128
    Likes Received:
    22
    Occupation:
    Internet Marketer/Actor
    Location:
    Oregon
    Hey guys,

    I know how to get URL's based on keywords, with laserURL. But is there any program that will scape all the page url's of a specific website?

    For ex.

    lets say I have a digital camera offer, and I wanted every single URL that bestbuy.com has about digital cameras.

    IS this possible, or is it something I need to get made? Any and all help is most appreciated, thanks.
     
  2. zorphee

    zorphee Registered Member

    Joined:
    May 16, 2009
    Messages:
    71
    Likes Received:
    6
    Occupation:
    What's this? Lol..
    Location:
    Singapore LAH.
    how does laserurl work? if it scrapes google results.. You can just type
    anyway, would be great if you can share the laserurl software. I'm using something else that only scrapes yahoo.. :(
     
  3. zorphee

    zorphee Registered Member

    Joined:
    May 16, 2009
    Messages:
    71
    Likes Received:
    6
    Occupation:
    What's this? Lol..
    Location:
    Singapore LAH.
    how does laserurl work? if it scrapes google results.. You can just type
    anyway, would be great if you can share the laserurl software. I'm using something else that only scrapes yahoo.. :(
     
  4. hivo22700

    hivo22700 Newbie

    Joined:
    Sep 22, 2008
    Messages:
    14
    Likes Received:
    1
    There are a couple of http://www.blackhatworld.com/blackhat-seo/member-downloads/110168-get-scraper-pack.html

    I'm new to coding and scraping. I would love to know how to scrape the URLs for an entire site and export them to a csv file.

    I basically want to scrape all the external links from one particular site to see where they are linking to. Does anybody know of any tools that could help me do this?
     
  5. revelation

    revelation Newbie

    Joined:
    May 22, 2009
    Messages:
    29
    Likes Received:
    597
    There's a Firefox add-on called Link Gopher I believe. Might be what you're looking for.
     
    • Thanks Thanks x 2
  6. nambooooo

    nambooooo Regular Member

    Joined:
    May 21, 2009
    Messages:
    219
    Likes Received:
    71
    Location:
    GreenMTN
    Try Web Data Extractor, you can find a cracked version around. It's pretty nice, you give the start URL and select what you wanna scrape off that site like URLs, emails, telephone nrs. Just play with the settings a little bit, but it's pretty intuitive.
     
    • Thanks Thanks x 2
  7. JohnKarr

    JohnKarr Junior Member

    Joined:
    Aug 12, 2009
    Messages:
    128
    Likes Received:
    22
    Occupation:
    Internet Marketer/Actor
    Location:
    Oregon
    It's in the forum just search for it



     
  8. hivo22700

    hivo22700 Newbie

    Joined:
    Sep 22, 2008
    Messages:
    14
    Likes Received:
    1
    @nambooooo - thanks that looks like exactly the type of tool I was looking for. I've had a look through the documentation, do you know if there is a way I can filter out "no follow" links from the results?

    @revelation, thanks - looks like a pretty good tool to quickly extract data.

    I think the Laser URL software is free anyway. http://www.mattharward.com/ who made it, he's got some other good tools on his site.
     
  9. nambooooo

    nambooooo Regular Member

    Joined:
    May 21, 2009
    Messages:
    219
    Likes Received:
    71
    Location:
    GreenMTN
    @hivo22700 no, Web Data Extractor is not gonna help you on that, it's just a scraper, not a SEO tool.
     
  10. techsmart

    techsmart Newbie

    Joined:
    Mar 3, 2009
    Messages:
    20
    Likes Received:
    0
    Last time I was actually looking for an automated scraping tool on google. I found Automation Anywhere, it has pretty nice capability to scrape data from websites. I have used it to scrape html tables, contents & even URL (as your case). You might want to try this http://www.automationanywhere.com/solutions/screenScrape.htm.
     
    Last edited: Jun 22, 2010
  11. risefromdeath

    risefromdeath Power Member

    Joined:
    Jul 1, 2009
    Messages:
    650
    Likes Received:
    107
    isnt it the wrong place for this thread?