1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

how to scrape protected page?

Discussion in 'C, C++, C#' started by extra, Dec 15, 2010.

  1. extra

    extra Jr. VIP Jr. VIP Premium Member

    Joined:
    May 18, 2007
    Messages:
    533
    Likes Received:
    34
    Hello
    I try to scrape password protected page.
    I'm using httpwebrequest/response
    but for soem reason it dont work. i try to store the cookies
    in container and use it but it doesnt work.
    Can anyone help me with source example for scraping protexted page in c sharp?

    Thanks in advance
     
  2. kill4

    kill4 Regular Member

    Joined:
    May 11, 2009
    Messages:
    243
    Likes Received:
    42
    Location:
    Texas
    Is it a page you have to login to see? if yes are you using the correct right login string example login=username&passwd=password .. Also some pages have hidden input fields that have values you have to include in the login string.. You would have to parse these values out in include them in the login string..

    I recommend using the firefox plugin Live HTTP Headers to capture the http login packet, If your not already using it.
     
  3. mindless

    mindless Junior Member

    Joined:
    Sep 5, 2010
    Messages:
    103
    Likes Received:
    3
    You can take software like teleport pro, which can be used to scrap protected pages but you should know atleast login and password