1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to Harvest all the Websites links. +REP AND THANKS

Discussion in 'Black Hat SEO' started by Silencer, Jan 29, 2010.

  1. Silencer

    Silencer Senior Member

    Joined:
    Dec 14, 2008
    Messages:
    1,149
    Likes Received:
    1,639
    I want to scrape

    Code:
    http://www.imdb.com/
    and scrape all the movie links, such as
    Code:
    http://www.imdb.com/title/tt1037705/
    How would I go upon doing that? I wasn't able to find a good url harvester.
     
  2. tabish8612

    tabish8612 Power Member

    Joined:
    Sep 13, 2009
    Messages:
    504
    Likes Received:
    109
    Occupation:
    Online Marketing
    Location:
    In your home...COMPUTER
    There is a firefox addon "Link Gopher". It will do this job for you. It will extract all the link, sort them and also remove duplicate.
    Another addon is "Multiple Links". It will allow you to select area and all link in the selected area will either copy in clipboard, open in new tabs or windows you have option to do as you like
     
    • Thanks Thanks x 1
  3. SEOHolicc

    SEOHolicc Newbie

    Joined:
    Jan 23, 2008
    Messages:
    33
    Likes Received:
    5
    Occupation:
    Internet Marketing
    Location:
    Colorado
    Home Page:
    Have you tried Xenu Link Sleuth? You can set it to grab internal and external links too. A site as big as IMDB would take a while though.
     
  4. RAKENSU

    RAKENSU Newbie

    Joined:
    May 10, 2009
    Messages:
    47
    Likes Received:
    33
    use httrack or webcopier ( i myself use webcopier ). Find it on phazeddl.c*m or just google it :)
     
  5. Hijinx

    Hijinx Junior Member

    Joined:
    Apr 13, 2009
    Messages:
    142
    Likes Received:
    87
    Location:
    New Jersey
    I found this plugin a while ago ...

    Code:
    http://wordpress.org/extend/plugins/imdb-link-transformer/
    To use it you basically just put the movie name in tags and it goes out and grabs the movie poster and some other data from imdb... it works nice, but does not pull the video...

    Code:
    In the same way, this plugin can display [B]many movie's related data inside a post[/B], when putting a movie name in [imdblt][/imdblt] tags. No widget needed, and movie's data can be displayed anywhere inside posts.
    I don't know enough php to do it, but i was thinking that if you use WPRobot and find a good RSS feed there might be a way to add the [imdblt] movie_Title [/imdblt] to the wp-robot posting template ...

    Hope it helps...
     
    • Thanks Thanks x 1
    Last edited: Jan 30, 2010
  6. vette09

    vette09 Junior Member

    Joined:
    Sep 28, 2009
    Messages:
    174
    Likes Received:
    40
    Occupation:
    Traveler
    Location:
    Earth - I travel full time.