1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to scrape a certain site?

Discussion in 'Black Hat SEO' started by undeterminederror, Jul 1, 2008.

  1. undeterminederror

    undeterminederror BANNED BANNED

    Joined:
    Mar 31, 2008
    Messages:
    630
    Likes Received:
    457
    I need to get all phrases with a specific keyword from a certain site. The site is too big to use google function site:thesite.com "keyword combination".
    There is any solution for that?
     
  2. goawayplease

    goawayplease Regular Member

    Joined:
    Apr 10, 2008
    Messages:
    299
    Likes Received:
    67
    Are you trying to get phrases from within the content, or get all the pages that contain "keyword combination?"
     
  3. brainet77

    brainet77 Junior Member

    Joined:
    Jun 10, 2008
    Messages:
    112
    Likes Received:
    35
    are you trying to scrape content from Search Engine or a regular sites?
     
  4. BozoClown

    BozoClown Junior Member

    Joined:
    Jan 4, 2009
    Messages:
    150
    Likes Received:
    106
    You could write a script(i would use perl) to get every keyword on the page, 10 words before it and 10 words after it. Then output as new line in a file. I guess you could do it in less than 15 lines of code.

    If I could write the script off the top of my head I would but I can't write an original line of code in perl. Despite that I have come up with perl scripts that do similar harvesting/mining just by loading dozens of mini-tutorials off google then mixing and matching to get what I want. Sounds complicated, but it is very easy to do, just find the first tutorial with code that does something even remotely related.
     
  5. undeterminederror

    undeterminederror BANNED BANNED

    Joined:
    Mar 31, 2008
    Messages:
    630
    Likes Received:
    457
    You are making fun of me? LOL. wtf is "line of code?" i never wanna hear about that again. ;)
    ... yes, to scrape a site, not a search engine. and all the phrases, not the pages :)
     
  6. ghands

    ghands Junior Member

    Joined:
    Feb 28, 2008
    Messages:
    101
    Likes Received:
    11
    Occupation:
    IT Technician
    Location:
    Dewsbury, Yorkshire, UK
    Hey Bozo, is there any reason that you would use Pearl versus PHP Curl. I use a WAMP server so testing is pretty easy.