1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HELP!? Multithreaded HttpWebRequests?

Discussion in 'Visual Basic .NET' started by MarketerX, Sep 4, 2011.

  1. MarketerX

    MarketerX Regular Member

    Joined:
    Mar 7, 2010
    Messages:
    398
    Likes Received:
    120
    Hey BHW, Googled a bit for this but kind of confused so hoping someone here can help me. If someone can explain this to me in detail, I will hook them up with whatever 1st bot I make (it'll be good ;) )

    Basically I want the user to be able to set the amount of threads in a textbox control, and they will all run parallel to each other.

    Someone has to know :)
     
  2. pirrtaste

    pirrtaste Junior Member

    Joined:
    Sep 1, 2011
    Messages:
    126
    Likes Received:
    40
    I have a example for you pm me, and I will send you code

    you can spawn threads using the code below
    Code:
    Dim T As New Thread(AddressOf GetWebsite)
    
                    T.Name = Website
                    T.Start()
    
    Than on the actual thread you can go

    Code:
    Thread.CurrentThread.Name
    
    To get the current website url and go from there , as i said pm me for full example
     
    • Thanks Thanks x 1
  3. darkmonk

    darkmonk Regular Member

    Joined:
    Nov 21, 2007
    Messages:
    226
    Likes Received:
    52
    Do you understand/know multithreaded .NET development and concurrent programming, and you just want to know how to wrap the HTTP requests, or do you need the foundational knowledge too? Do you know how to write a thread safe singleton, locking, etc? What language? Do you know how to communicate back from your threads to the UI using BeginInvoke, etc.?
     
  4. darkmonk

    darkmonk Regular Member

    Joined:
    Nov 21, 2007
    Messages:
    226
    Likes Received:
    52
    Nevermind, didn't look at the subforum above, so...it's VB.NET. I know C# and VB.Net and have done a ton of multithreaded .NET dev. Need some idea of where you are starting from knowledge-wise.
     
  5. MarketerX

    MarketerX Regular Member

    Joined:
    Mar 7, 2010
    Messages:
    398
    Likes Received:
    120
    Thanks for the response darkmonk.

    My VB.NET knowledge is beginner-intermediate, I have read a few tutorials, watched a few training videos. I understand that to make the type of bots I need, I will want the following features...(not going to list them all, just the major ones I need help with if possible.)

    1. Multithreading
    2. Proxy Support
    3. Decaptcher API Support

    I also understand that to create most HttpWebRequest bots, I need to go to the website that I want to automate and analyze the POST data being sent...correct?
     
  6. darkmonk

    darkmonk Regular Member

    Joined:
    Nov 21, 2007
    Messages:
    226
    Likes Received:
    52
    Yes, you'll need to analyze the form and figure out what to post back.

    Ok, so beginner knowledge and you want to do a multithreaded app... You don't necessarily _need_ but I would recommend that you design in some way for your threads to report progress back to the main app. Depending on how sophisticated of a system you are coding, you might go to the next level and design your threads to deal with work units, so you can maintain a thread pool with a queue for work units and a class that doles out work units to available threads. However, you might not need to get that fancy.

    If all you are building is a Windows.Forms app with a form to automate a single target website, and you want to be able to kick off a configurable number of threads that run the same thing against that one site...then it is pretty easy. But what about the data that the threads are using to populate the form? You are going to need some way to manage that and distribute it to the threads.

    Are you using .NET 4.0, .NET 3.5 or an older platform?

    Some resources:

    Joe Duffy's blog is handy, his book is a good resource too:
    Code:
    http://www.bluebytesoftware.com/blog/Default.aspx
    MSDN page on the Thread class:
    Code:
    http://msdn.microsoft.com/en-us/library/system.threading.thread.aspx
    If I were doing it, I would build:

    - Write a Thread safe generic Queue<T> (or Queue(Of T) in vb.net), the out of the box .NET generics are not thread safe, you have to implement it yourself, you can find examples out there but its pretty easy to wrap it and then implement locking around enqueue/dequeue/peek/etc.
    - Define an interface and a class representing a set of data to populate (username, email address, etc. etc.)
    - Define an interface and a class representing a target site's form you are automating, I would write this in a generic way so that you can deal with scanning and dynamically adapting to different forms
    - Define an interface and a class representing a work unit (a post to a target site), which would contain the above two classes
    - Define and build a thread manager class that polls the queue for work units and distributes them to available threads. The thread manager receives progress updates from the worker threads and then relays the progress info to the UI (instead of coding the UI updates into your thread implementations although you could do it that way too, but what if down the road you want to wrap your threading/work unit processing into a windows service etc.)
    - Your worker thread receives the work unit from the thread manager, figures out what to do with it, executes your HttpWebRequest(s), reports progress and results back to the manager/coordinator class (e.g. orchestrator)

    Your worker threads execute a delegate. You write the delegate and then when the thread is constructed you tell it what delegate to run and you can pass it a state object. Writing the delegate is just like writing a function. But remember, as soon as you go multi-threaded you have to be careful about sharing variables etc. and you need to lock access to them. The way I usually do it is with another class that stores the shared data with the manager class brokering access to it via readerwriter locks (this way you can have many readers without slowing things down).

    If this sounds like its overcomplicated or is exploding into a lot of work - it is a lot of work. But you can't avoid it. If you try to take shortcuts, you WILL run into mysterious threading errors, deadlocks, race conditions, etc. etc. You have to do it right up front if you want it to work reliably without blowing up.

    So now that I made your head explode with the above, I will tell you the easy way to just jam it through if you want to bang it out.... use BackgroundWorker:

    Code:
    http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx
    There's a bunch of VB.NET examples on that page. I never do it this way myself, but for a simple app it will work. BackgroundWorker is easier because they've got a lot of the UI interaction figured out for you, but there is an upper bound to how many you can create.
     
    • Thanks Thanks x 3
  7. Hydrogen

    Hydrogen Newbie

    Joined:
    Dec 30, 2009
    Messages:
    39
    Likes Received:
    23
    Occupation:
    Co-Owner of AdvertMarketing
    Home Page:
    Learn Classes my friend and each thread load your CLASS, by doing this you can put all the 'actions' that a 'WORKER THREAD' can or should be doing all included in that class so you don't have to worry about too many x-thread communications. (Which you still will need to do to update any UI element from your spawned threads)

    You have a few options here, either use Background Workers or create your own threads, exceptions, delegates...etc. Starting out you should probably get real intimate knowledge of background workers and work your way up from there.

    Also wrap try/catch statements around code that you think 'MAY error out or time out' (ie. httpwebrequest calls) the reason being if you do not try any errors and there is a issue with a website you are communicating with it will crash your app, and it may take you a while to figure out WTF is going on and why randomly your app crashes. (I Suggest when a Error is Caught to use a delgate to the main form to write the ex (error statement to a error.log file so you can easily go back and debug and more in depthly have your app 'WATCH and HANDLE' these particular errors individually.