C#: How to multithread httpwebrequests?
- Thread starter kytro360
- Start date
machines-in-motion
Junior Member
- Aug 11, 2010
- 129
- 46
Make a method that does the httpwebrequest in a new object type (referred to here as RequestObj), and for each time you want to use it, spawn a new object/thread with:
You can keep the results in the object for future reference, if you need them, or just use yourobject.Dispose(); at the end.
EDIT: I forgot to mention, you'll need:
at the beginning.
Code:
RequestObj yourobject = new RequestObj();
Thread thread = new Thread(new ThreadStart((yourobject.method));
thread.Start();EDIT: I forgot to mention, you'll need:
Code:
using System;
using System.Threading;
Last edited:
accelerator_dd
Elite Member
- May 14, 2010
- 2,447
- 1,022
First you need to include the namespace:
Then you make a function that accepts an object:
then in the button even u want to make it start (adding threads and working)
then use:
Do this for each task u want and they will be running simontanuously.
Cheers, wish u all the best
p.s. Threads can be a headache if you need a bit more advanced stuff.
Code:
using System.Multithreading;Then you make a function that accepts an object:
Code:
void YourFunctName(object argument1)
{
//code here
}then in the button even u want to make it start (adding threads and working)
then use:
Code:
ThreadPool.QueueUserWorkItem(YourFunctName, argument1);Do this for each task u want and they will be running simontanuously.
Cheers, wish u all the best
p.s. Threads can be a headache if you need a bit more advanced stuff.
accelerator_dd
Elite Member
- May 14, 2010
- 2,447
- 1,022
Put the code in the YourFunctName and declare it there as well.... each will have a local version, so not a problem if they have the same name. Scopes of variables
.After taht for adding more threads also use a for loop to add as many as u want.
xenon2010
Regular Member
- Apr 27, 2010
- 233
- 49
its very bad idea to use new threads... its very hard to control them and track them..
the best way I usually use is to use the httpwebrequest inside "Parallel.ForEach" loop..
easy to handle and way way way faster and better... and you can run as much threads as you want...
the best way I usually use is to use the httpwebrequest inside "Parallel.ForEach" loop..
easy to handle and way way way faster and better... and you can run as much threads as you want...
xenon2010
Regular Member
- Apr 27, 2010
- 233
- 49
well i'm not gonna post whole codes, just simple one just so you can get the idea:
anyway, the loop will run 2 threads at a time. once a thread exits another one will be lunched till the loop go through all of the links inside the list..
hope this is enough for you to get the idea.. if you need more help use google as there are plenty of examples and tutorials of how to do this..
rep me up
PHP:
//list of urls you want to run in parallel
List<string> URLsList = new List<string>();
URLsList.Add("http://blabla.com");
URLsList.Add("http://blabla2.com");
URLsList.Add("http://blabla3.com");
URLsList.Add("http://blabla4.com");
URLsList.Add("http://blabla5.com");
//..... etc
Parallel.ForEach(URLsList, new ParallelOptions() { MaxDegreeOfParallelism = 2 },
(url, i, j) =>
{
//url = the url of the current thread
//j = thread number..
//==========================
//add your webclient request here...
//and do other processing etc...
});anyway, the loop will run 2 threads at a time. once a thread exits another one will be lunched till the loop go through all of the links inside the list..
hope this is enough for you to get the idea.. if you need more help use google as there are plenty of examples and tutorials of how to do this..
rep me up
this code worked great for me.
First you need to include the namespace:
Code:using System.Multithreading;
Then you make a function that accepts an object:
Code:void YourFunctName(object argument1) { //code here }
then in the button even u want to make it start (adding threads and working)
then use:
Code:ThreadPool.QueueUserWorkItem(YourFunctName, argument1);
Do this for each task u want and they will be running simontanuously.
Cheers, wish u all the best
p.s. Threads can be a headache if you need a bit more advanced stuff.
botrockets
Regular Member
- Mar 16, 2013
- 389
- 647
List<string> list = new List<string>();
list.Add("url1");
list.Add("url2");
list.Add("url3");
foreach (string link in list)
{
HttpWebRequest req = WebRequest.Create(link) as HttpWebRequest;
if (req != null)
{
req.BeginGetResponse(
ar =>
{
var r = ar.AsyncState as HttpWebRequest;
if (r != null)
{
var response = (HttpWebResponse)r.EndGetResponse(ar);
// do anything with response...
}
}, req);
}
}
list.Add("url1");
list.Add("url2");
list.Add("url3");
foreach (string link in list)
{
HttpWebRequest req = WebRequest.Create(link) as HttpWebRequest;
if (req != null)
{
req.BeginGetResponse(
ar =>
{
var r = ar.AsyncState as HttpWebRequest;
if (r != null)
{
var response = (HttpWebResponse)r.EndGetResponse(ar);
// do anything with response...
}
}, req);
}
}
rootjazz
Elite Member
- Dec 21, 2012
- 3,166
- 1,190
Async aside, as OP asked about threading an HttpRequest.
You should look into the Parallel functionality provider in .net 4.0+ as that is quite useful (which is the same as tasks I believe, but I don't personally use tasks much).
Parallel is good as .net handles a lot of the complicated stuff for you, such as decided how many threads to run for max efficiency.
Another way would be to look into the blocking collections / queue if you want to control how many threads are to run. But this will require some work on your behalf to get it ready
Or just if you want to fire threads, do note you will need to manage how many threads are running, when to pause / sleep / block otherwise you are just going to start a load of threads and chock your processor / processors
You should look into the Parallel functionality provider in .net 4.0+ as that is quite useful (which is the same as tasks I believe, but I don't personally use tasks much).
Parallel is good as .net handles a lot of the complicated stuff for you, such as decided how many threads to run for max efficiency.
Code:
Parallel.Foreach(listUrls,url=>{
//do what you need to do on the URL
//eg webBot.GetPage(url);
});Another way would be to look into the blocking collections / queue if you want to control how many threads are to run. But this will require some work on your behalf to get it ready
Or just if you want to fire threads, do note you will need to manage how many threads are running, when to pause / sleep / block otherwise you are just going to start a load of threads and chock your processor / processors
Code:
foreach(var url in listUrls){
var thread = new Thread ( ()=>{
//do your work here
});
thread.isBackground = true;
thread.start();
}
tahworld
Regular Member
- Aug 16, 2013
- 456
- 397
Sorry to bump this old thread, but I didn't want to make a new one, because I have the exact same question.
The above codes work really well, so now the next step. How to incorporate rotating proxies into this?
Would you use some sort of refresh method with random proxy assignment?
The above codes work really well, so now the next step. How to incorporate rotating proxies into this?
Would you use some sort of refresh method with random proxy assignment?