1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

need code for harvesting url from search engines.. please help...

Discussion in 'C, C++, C#' started by blackhattrial, Sep 29, 2009.

  1. blackhattrial

    blackhattrial Regular Member

    Joined:
    Jul 22, 2008
    Messages:
    410
    Likes Received:
    131
    Location:
    black hat world -
    hello this is a request for a piece of code to harvest urls from search engines.

    +rep 2 all those who help me...
     
  2. pavan_buzz

    pavan_buzz Junior Member

    Joined:
    Jul 25, 2009
    Messages:
    169
    Likes Received:
    115
    Occupation:
    Freelancer
    Location:
    Internet
    Hey mate URL's like wat ??
     
  3. xhpdx

    xhpdx Regular Member

    Joined:
    Sep 21, 2008
    Messages:
    331
    Likes Received:
    2,160
    Occupation:
    Coder
    Location:
    EU
    you want to scrape the serps or what ?
     
  4. dparker

    dparker Newbie

    Joined:
    Feb 7, 2009
    Messages:
    28
    Likes Received:
    100
    Occupation:
    Software enginner
    Location:
    Canada
    1. Scrape web page.
    2. Use regex to extract urls.


    This regex should get you going

    @"((https?|ftp|gopher|telnet|file|notes|ms-help):((//)|(\\\\))+[\w\d:#@%/;$()~_?\+-=\\\.&]*)"

    Scraping a page is fairly simple. Just search the web for C# web scrape or curl if you are using php.
     
    • Thanks Thanks x 1
  5. blackhattrial

    blackhattrial Regular Member

    Joined:
    Jul 22, 2008
    Messages:
    410
    Likes Received:
    131
    Location:
    black hat world -
    @dparker
    thanks rep added


    wld do it in c# have to learn everything...

    thanks a ton