1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

need to scrape a complete site. tool needed

Discussion in 'BlackHat Lounge' started by theconjurer, May 10, 2009.

  1. theconjurer

    theconjurer Junior Member

    Joined:
    Dec 22, 2008
    Messages:
    187
    Likes Received:
    301
    Occupation:
    keep guessing...
    Location:
    the Internet...
    Hello everyone.

    I need a tool that can scrape a complete site. I dont want to publish it on another site directly. Just need a tool that can scrape a site (not a forum) and download all the scraped pages in either .txt or .doc or .rtf format and save it on my system.

    Is there a tool that is already available that can do the same? or do I need to get it coded?

    thanks.

    P.S. for the inquisitive, I am building an information site and therefore need to get all available information from a couple of different sites...download it to my system and then sort it according to my own parameters and then upload it later. I just need a tool for scraping the content. cheers.
     
  2. sikx

    sikx Registered Member

    Joined:
    Jan 4, 2009
    Messages:
    65
    Likes Received:
    166
    Location:
    Germany
    Home Page:
  3. theconjurer

    theconjurer Junior Member

    Joined:
    Dec 22, 2008
    Messages:
    187
    Likes Received:
    301
    Occupation:
    keep guessing...
    Location:
    the Internet...
    site does not open. getting a page load error. thanks for the effort mate

    edit: either that, or I don't know how to follow your reply. thanks.
     
    Last edited: May 10, 2009
  4. sikx

    sikx Registered Member

    Joined:
    Jan 4, 2009
    Messages:
    65
    Likes Received:
    166
    Location:
    Germany
    Home Page:
  5. venusweb

    venusweb Newbie

    Joined:
    Apr 20, 2009
    Messages:
    5
    Likes Received:
    1
    Occupation:
    Freelance Software Engineer
    Location:
    New Delhi, India
    Google "torrent site ripper" you will get what you want or PM me the URL and I will be able to give a better advice.
     
    • Thanks Thanks x 1
  6. 94FBR

    94FBR Registered Member

    Joined:
    Feb 25, 2009
    Messages:
    67
    Likes Received:
    24
    • Thanks Thanks x 1
    Last edited: May 10, 2009
  7. theconjurer

    theconjurer Junior Member

    Joined:
    Dec 22, 2008
    Messages:
    187
    Likes Received:
    301
    Occupation:
    keep guessing...
    Location:
    the Internet...
    thanks for the additional info. I am personally looking for something else. I must admit that it is something i've never heard off. I did go through the wiki and the documentation but hey, the web copier tool seems like an easier option.

    found torrent site ripper. thanks. What's the URL that you want btw?

    downloaded and scanned. Kaspersky internet security 2009 did not show any threats. Testing it now. thanks mate.
     
  8. jerra

    jerra Regular Member

    Joined:
    Sep 18, 2008
    Messages:
    476
    Likes Received:
    64
    Home Page:
    Will this copy and flash videos that are embedded in a site also do you know?
     
  9. blackhaty

    blackhaty Junior Member

    Joined:
    Feb 24, 2008
    Messages:
    125
    Likes Received:
    50
    Yes most of website rippers will copy flash content also.
     
  10. SEO20

    SEO20 Elite Member

    Joined:
    Mar 25, 2009
    Messages:
    2,017
    Likes Received:
    2,259
    wget is freaking awesome. Playing a lot with it these days

     
  11. ukescuba

    ukescuba Jr. VIP Jr. VIP Premium Member

    Joined:
    Feb 24, 2008
    Messages:
    994
    Likes Received:
    634
    Occupation:
    Mobile Marketer & QR Code Junkie
    Location:
    San Antonio, TX
    Home Page:
    depends how the flash video is embedded if its streamed using progressive download then chances are yes it will download this is probably the most common method of streaming video, however if the sites using real streaming video then usually you wont be able too

    but it maybe possible to hotlink to the streaming video unless the streaming server has the video/domain locked down

    also if the sites dynamic you may loose some its functionality

    hth
     
    Last edited: Jun 17, 2009