1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

C#: How to multithread httpwebrequests?

Discussion in 'C, C++, C#' started by kytro360, Sep 5, 2011.

  1. kytro360

    kytro360 Power Member

    Joined:
    Jan 12, 2010
    Messages:
    703
    Likes Received:
    733
    Hey, I am a newb to multithreading in general. I want to multithread a httpwebrequest I am running so I can run more than one of it at a time. How would I go about doing this?
     
  2. machines-in-motion

    machines-in-motion Junior Member

    Joined:
    Aug 11, 2010
    Messages:
    129
    Likes Received:
    46
    Occupation:
    Engineer
    Location:
    Here and there
    Make a method that does the httpwebrequest in a new object type (referred to here as RequestObj), and for each time you want to use it, spawn a new object/thread with:
    Code:
    RequestObj yourobject = new RequestObj();
    Thread thread = new Thread(new ThreadStart((yourobject.method));
    thread.Start();
    You can keep the results in the object for future reference, if you need them, or just use yourobject.Dispose(); at the end.

    EDIT: I forgot to mention, you'll need:
    Code:
    using System;
    using System.Threading;
    at the beginning.
     
    Last edited: Sep 5, 2011
  3. kytro360

    kytro360 Power Member

    Joined:
    Jan 12, 2010
    Messages:
    703
    Likes Received:
    733
    Ok lets say you call my method by doing this webrequest (). What would be the code to call it multiple times?
     
  4. kytro360

    kytro360 Power Member

    Joined:
    Jan 12, 2010
    Messages:
    703
    Likes Received:
    733
    double post.....
     
    Last edited: Sep 6, 2011
  5. kytro360

    kytro360 Power Member

    Joined:
    Jan 12, 2010
    Messages:
    703
    Likes Received:
    733
    triple post...
     
    Last edited: Sep 6, 2011
  6. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,441
    Likes Received:
    1,005
    Occupation:
    SEO
    Location:
    IM Wonderland
    First you need to include the namespace:

    Code:
    using System.Multithreading;
    Then you make a function that accepts an object:

    Code:
    void YourFunctName(object argument1)
    {
    //code here
    }
    then in the button even u want to make it start (adding threads and working)

    then use:
    Code:
    ThreadPool.QueueUserWorkItem(YourFunctName, argument1);
    Do this for each task u want and they will be running simontanuously.

    Cheers, wish u all the best :)

    p.s. Threads can be a headache if you need a bit more advanced stuff.
     
  7. kytro360

    kytro360 Power Member

    Joined:
    Jan 12, 2010
    Messages:
    703
    Likes Received:
    733
    Hey I see your code lets you multitask, whats the code to run several of the same httpwebrequests in multithreaded?
     
  8. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP Premium Member

    Joined:
    May 14, 2010
    Messages:
    2,441
    Likes Received:
    1,005
    Occupation:
    SEO
    Location:
    IM Wonderland
    Put the code in the YourFunctName and declare it there as well.... each will have a local version, so not a problem if they have the same name. Scopes of variables ;).


    After taht for adding more threads also use a for loop to add as many as u want.
     
  9. xenon2010

    xenon2010 Regular Member

    Joined:
    Apr 27, 2010
    Messages:
    231
    Likes Received:
    48
    Occupation:
    web and desktop apps programmer
    Location:
    prison
    Home Page:
    its very bad idea to use new threads... its very hard to control them and track them..
    the best way I usually use is to use the httpwebrequest inside "Parallel.ForEach" loop..
    easy to handle and way way way faster and better... and you can run as much threads as you want...
     
  10. kytro360

    kytro360 Power Member

    Joined:
    Jan 12, 2010
    Messages:
    703
    Likes Received:
    733
    Any code? Please note Im new to multithreading
     
  11. xenon2010

    xenon2010 Regular Member

    Joined:
    Apr 27, 2010
    Messages:
    231
    Likes Received:
    48
    Occupation:
    web and desktop apps programmer
    Location:
    prison
    Home Page:
    well i'm not gonna post whole codes, just simple one just so you can get the idea:
    PHP:
    //list of urls you want to run in parallel
    List<stringURLsList = new List<string>();
    URLsList.Add("http://blabla.com");
    URLsList.Add("http://blabla2.com");
    URLsList.Add("http://blabla3.com");
    URLsList.Add("http://blabla4.com");
    URLsList.Add("http://blabla5.com");
    //..... etc

               
    Parallel.ForEach(URLsList, new ParallelOptions() { MaxDegreeOfParallelism },
                                 (
    urlij) =>
                                 {
        
    //url = the url of the current thread
        //j = thread number..
        //==========================
        //add your webclient request here...
        //and do other processing etc...
        
    });
    anyway, the loop will run 2 threads at a time. once a thread exits another one will be lunched till the loop go through all of the links inside the list..
    hope this is enough for you to get the idea.. if you need more help use google as there are plenty of examples and tutorials of how to do this..

    rep me up :D
     
    • Thanks Thanks x 4
  12. kytro360

    kytro360 Power Member

    Joined:
    Jan 12, 2010
    Messages:
    703
    Likes Received:
    733
    Your codes in PHP
     
  13. xenon2010

    xenon2010 Regular Member

    Joined:
    Apr 27, 2010
    Messages:
    231
    Likes Received:
    48
    Occupation:
    web and desktop apps programmer
    Location:
    prison
    Home Page:
    nope its not...
    I just wrapped the codes with
    PHP:
     tags...
     
  14. ninja208

    ninja208 Newbie

    Joined:
    Jul 12, 2013
    Messages:
    21
    Likes Received:
    1
    this code worked great for me.
     
  15. sm754

    sm754 Registered Member

    Joined:
    Mar 21, 2012
    Messages:
    93
    Likes Received:
    38
    Occupation:
    Farmer
    Location:
    Azerbaijan
    Make sure to limit the amount of requests you have going at once - you could crash a weak router by making a few hundred requests simultaneously.
    If you're using ThreadPool, you can use the SetMaxThreads function for this without bothering too much.
     
  16. Piotr__11

    Piotr__11 Junior Member

    Joined:
    Mar 13, 2012
    Messages:
    122
    Likes Received:
    69
    Remember to increase the DefaultConnectionLimit
    I spent hours on thinking why my multithreaded app work slower than a few single threaded apps running at once.
     
  17. botrockets

    botrockets Regular Member

    Joined:
    Mar 16, 2013
    Messages:
    272
    Likes Received:
    463
    Occupation:
    Software Developer
    Location:
    Saint T.N.
    List<string> list = new List<string>();


    list.Add("url1");
    list.Add("url2");
    list.Add("url3");


    foreach (string link in list)
    {
    HttpWebRequest req = WebRequest.Create(link) as HttpWebRequest;
    if (req != null)
    {
    req.BeginGetResponse(
    ar =>
    {
    var r = ar.AsyncState as HttpWebRequest;
    if (r != null)
    {
    var response = (HttpWebResponse)r.EndGetResponse(ar);


    // do anything with response...
    }
    }, req);
    }
    }
     
  18. kingc0der

    kingc0der Newbie

    Joined:
    Jun 1, 2013
    Messages:
    4
    Likes Received:
    2
    Occupation:
    Software Dev/Programmer
    You dont need to use threading. Create a class file with your HttpRequest function in Async format, the load your class file in a garbage collection on your main forum.
     
  19. rootjazz

    rootjazz Jr. VIP Jr. VIP

    Joined:
    Dec 21, 2012
    Messages:
    617
    Likes Received:
    313
    Occupation:
    Developer
    Location:
    UK
    Home Page:
    Async aside, as OP asked about threading an HttpRequest.

    You should look into the Parallel functionality provider in .net 4.0+ as that is quite useful (which is the same as tasks I believe, but I don't personally use tasks much).

    Parallel is good as .net handles a lot of the complicated stuff for you, such as decided how many threads to run for max efficiency.

    Code:
    Parallel.Foreach(listUrls,url=>{
       //do what you need to do on the URL
       //eg   webBot.GetPage(url);
    });
    

    Another way would be to look into the blocking collections / queue if you want to control how many threads are to run. But this will require some work on your behalf to get it ready

    Or just if you want to fire threads, do note you will need to manage how many threads are running, when to pause / sleep / block otherwise you are just going to start a load of threads and chock your processor / processors
    Code:
    foreach(var url in listUrls){
       var thread = new Thread ( ()=>{
          //do your work here
    
    
       });
       thread.isBackground = true;
       thread.start();
    }
    
    
     
  20. tahworld

    tahworld Regular Member

    Joined:
    Aug 16, 2013
    Messages:
    457
    Likes Received:
    393
    Location:
    ✔✔✔✔✔✔✔
    Sorry to bump this old thread, but I didn't want to make a new one, because I have the exact same question.

    The above codes work really well, so now the next step. How to incorporate rotating proxies into this?

    Would you use some sort of refresh method with random proxy assignment?