need code for harvesting url from search engines.. please help...

Discussion in 'C, C++, C#' started by blackhattrial, Sep 29, 2009.

  1. blackhattrial

    blackhattrial Regular Member

    Joined:
    Jul 22, 2008
    Messages:
    410
    Likes Received:
    131
    Location:
    black hat world -
    hello this is a request for a piece of code to harvest urls from search engines.

    +rep 2 all those who help me...
     
  2. pavan_buzz

    pavan_buzz Junior Member

    Joined:
    Jul 25, 2009
    Messages:
    169
    Likes Received:
    115
    Occupation:
    Freelancer
    Location:
    Internet
    Hey mate URL's like wat ??
     
  3. xhpdx

    xhpdx Regular Member

    Joined:
    Sep 21, 2008
    Messages:
    331
    Likes Received:
    2,160
    Occupation:
    Coder
    Location:
    EU
    you want to scrape the serps or what ?
     
  4. dparker

    dparker Newbie

    Joined:
    Feb 7, 2009
    Messages:
    28
    Likes Received:
    100
    Occupation:
    Software enginner
    Location:
    Canada
    1. Scrape web page.
    2. Use regex to extract urls.


    This regex should get you going

    @"((https?|ftp|gopher|telnet|file|notes|ms-help):weep://)|(\\\\))+[\w\d:#@%/;$()~_?\+-=\\\.&]*)"

    Scraping a page is fairly simple. Just search the web for C# web scrape or curl if you are using php.
     
    • Thanks Thanks x 1
  5. blackhattrial

    blackhattrial Regular Member

    Joined:
    Jul 22, 2008
    Messages:
    410
    Likes Received:
    131
    Location:
    black hat world -
    @dparker
    thanks rep added


    wld do it in c# have to learn everything...

    thanks a ton