1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Direction needed. I want to build a Semi-automatic Web browser in C#

Discussion in 'C, C++, C#' started by marcelomusza, Nov 2, 2012.

  1. marcelomusza

    marcelomusza Junior Member

    Joined:
    Oct 18, 2012
    Messages:
    198
    Likes Received:
    59
    Occupation:
    Web Developer / IMer
    Location:
    Argentina
    Hi guys,

    I am a Web Developer, so I have solid knowledge in C#.
    As I am working in my IM business like crazy, I want to create a semi-automated software to use several accounts to view and comment YT Videos manually, just by hand, selecting users from a list.

    I know how I am going to build this piece of software, but I never made this kind of tools before. I would like to hear your suggestions about where to start, is there a manual or tutorial around to educate myself about how to create a web browser kind of software?
    I am googling this but I am finding very random approaches, and I come here because I am sure you can easily share with me a more targeted approach.

    Should I use the WebBrowser class for this?

    Thanks in advance! every piece of advice will be much appreciated!
     
  2. collywobbles99

    collywobbles99 Junior Member

    Joined:
    Mar 22, 2009
    Messages:
    173
    Likes Received:
    21
    Location:
    UK
    Simplest is to use WebBrowser. It has multithreaded issues and also can sometimes be fooled as to when the page has actually loaded but it's the simplest to get going with.
     
  3. Mutikasa

    Mutikasa Power Member

    Joined:
    May 23, 2011
    Messages:
    579
    Likes Received:
    207
    I dont know about views, but commenting you can make with Youtube API.
     
  4. sn2100

    sn2100 Junior Member

    Joined:
    Mar 27, 2012
    Messages:
    128
    Likes Received:
    20
    gud luck ! Hope you do share your bot with BHW .. !
     
  5. balamut

    balamut Junior Member

    Joined:
    Aug 26, 2012
    Messages:
    105
    Likes Received:
    27
    Occupation:
    Firing People
    Location:
    Los Angeles
    WebBrowser.

    Just whatever you do, do it slow, make it act like a real human. YT keep changing things around and ever the real comments getting marked as a spam, I'm not even talking about captcha on the comments.

    Good luck mate
     
  6. blakamia

    blakamia Junior Member

    Joined:
    Jan 25, 2010
    Messages:
    162
    Likes Received:
    348
    Use WatiN. You can tie it to a WebBrowser control as well.
     
  7. funkseo

    funkseo Newbie

    Joined:
    Sep 27, 2012
    Messages:
    20
    Likes Received:
    1
    Webbrowser control is better, you can use WebClient or HttpWebResponse but it is not easy to post data.
    Webbrowser is slowest but it is not different too much from Internet Explorer.
     
  8. JOBGG

    JOBGG Newbie

    Joined:
    Oct 19, 2012
    Messages:
    17
    Likes Received:
    2
    The WebBrowser controll is essentially Internet Explorer wrapped up.
     
  9. eried

    eried Newbie

    Joined:
    Jan 17, 2013
    Messages:
    14
    Likes Received:
    10
    Location:
    Chile
    Home Page:
    There is a big problem with the WebBrowser: bad proxy support & no file upload :/
     
  10. BlackHatWhiteHearth

    BlackHatWhiteHearth BANNED BANNED

    Joined:
    Dec 26, 2012
    Messages:
    24
    Likes Received:
    9
    Forget about WebBrowser, WatiN
    First approach is to use WebClient, you must inherit the class to add additional functionality
    Code:
        using System;
        using System.Net;
        using System.Net.Security;
        using System.Security.Cryptography.X509Certificates;
        public class WebClientEx : WebClient
        {
            static WebClientEx()
            {
                ServicePointManager.Expect100Continue = false;
                //Ignore certificate errors
                ServicePointManager.ServerCertificateValidationCallback = (RemoteCertificateValidationCallback)((object sender,
                X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) => true);
                //This line is very important for multithreading
                //by default you can have up to 2 simultaneous connections, no matter how many threads you are creating
                ServicePointManager.DefaultConnectionLimit = 1000;
            }
    
    
            public WebClientEx()
                : base()
            {
                this.AllowAutoRedirect = true;
                this.Cookies = new CookieContainer();
            }
            
            protected override WebRequest GetWebRequest(Uri address)
            {
                //You can add other headers too
                HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
                if (request.Method == "POST")
                {
                    request.ContentType = "application/x-www-form-urlencoded";
                }
                //the webclient will handle GZip and Defalte compressions automatically, improves the performance a lot
                request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
                request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12";
                request.CookieContainer = this.Cookies;
                request.Timeout = 10000;
                return request;
            }
            
            //Add cookie support
            public CookieContainer Cookies { get; set; }
        }
    
    Basically WebClient is wrapping WebRequest and WebResponse classes and simplifies the things.
    Code:
    WebClientEx web = new WebClientEx();
    string htmlGet = web.DownloadString("yoururl");
    string htmlPost = web.UploadString("yoururl", "yourpostdata");
    byte[] dataGet = web.DownloadData("yoururl");
    byte[] dataPost = web.UploadData("yoururl", Encoding.UTF8.GetBytes("yourpostdata"));
    
    However you have no multipart-formdata support.

    The second approach is to use HttpClient, which is better and have multipart-formdata support. It is in .NET 4.5, but you can use it in VS 2010 and .NET 4.0
    Get it from NuGet:
    Code:
    PM> Install-Package Microsoft.Net.Http
    However, the latest version of HttpClient has no synchronous methods.
    I am using the older version with the sync methods, i am not sure why MS forced us to use only async methods.
    Get it from NuGet:
    Code:
    PM> Install-Package HttpClient -Version 0.3.0
    Or
    Code:
    PM> Install-Package HttpClient -Version 0.5.0
    - this is the lastest version that still has sync methods.

    Fiddler is a very helpful tool to intercept and monitor HTTP traffic, it is also free. Google it, i cannot post links.
     
    Last edited: Jan 18, 2013
  11. eried

    eried Newbie

    Joined:
    Jan 17, 2013
    Messages:
    14
    Likes Received:
    10
    Location:
    Chile
    Home Page:
    Doing anything with the WebClient object is a PITA, I think for complex bots you need another tool, another environment. I have sold bots using both techniques and WebBrowser is crappier but easier, WebClient is much more complete, but very lengthy to do something with it.
     
    Last edited: Jan 18, 2013
  12. ikstob

    ikstob Junior Member

    Joined:
    Nov 12, 2012
    Messages:
    147
    Likes Received:
    130
    Location:
    ikstob.com
    Home Page:
    Use Selenium (http://seleniumhq.org/) it is completely free / open-source, depending on your needs you can use their embedded browser, Firefox, Chrome (and I believe IE *shudder*). There are .NET bindings and its easy to use from C#. You have full DOM access to the loaded document, you can inject JavaScript instructions to the pages to simplify automation instead of trying to navigate the DOM and click links/buttons.

    I'm using the Java bindings with Selenium for all my IM bots, I can deploy them on cheap linux VPS's for a few cents/hour.
     
    • Thanks Thanks x 2
  13. eried

    eried Newbie

    Joined:
    Jan 17, 2013
    Messages:
    14
    Likes Received:
    10
    Location:
    Chile
    Home Page:
    Can you upload files with Selenium, into the filefield? That will be the full solution if is possible
     
  14. BlackHatWhiteHearth

    BlackHatWhiteHearth BANNED BANNED

    Joined:
    Dec 26, 2012
    Messages:
    24
    Likes Received:
    9
    Any browser automation is slow as hell and it is not easier coding a bot with browser than with just requests.
    And how about mutlithreading ? You are going to create hundreds of browser objects, sure this is a good idea.
    I hope the client will be satisfied with a program that consumes few GBs of RAM.
     
  15. 312k1t

    312k1t Registered Member

    Joined:
    Oct 26, 2012
    Messages:
    59
    Likes Received:
    6
    You can comment with Youtube API. Google has SDK for .NET. But make sure that you are using proxies and rotate several api keys. Or you get ban.
     
  16. terebi

    terebi Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 11, 2011
    Messages:
    357
    Likes Received:
    94
    Webclient is one way to do it, however you have to realize that a wc won't process javascript for you. Sites that really heavily on jscript won't work if you try to drive it using low level post/get.

    Selenium is a good way to go, however its not headless. IE a browser window is always open.
     
  17. zysfryar

    zysfryar Newbie

    Joined:
    Dec 6, 2012
    Messages:
    29
    Likes Received:
    7
    Thanks Ikstob, I've been trying to play with Selenium to replace a current Firefox/iMacros based setup.
    I hope it will prove leaner than my existing setup.