C#: How to multithread httpwebrequests?

Sorry to bump this old thread, but I didn't want to make a new one, because I have the exact same question.

The above codes work really well, so now the next step. How to incorporate rotating proxies into this?

Would you use some sort of refresh method with random proxy assignment?

I probably should not, but this is my copyrighted code. It should give you an idea of one method and does not address your exact question; there are many different methods and this should give you one idea - an array of threads.

After creating an array of threads, you loop through the thread array until one is found that is not busy. Once a not busy thread array is found, you load the object into that array bucket and call RunWorkerAsync[object array].

Code:
        /// <summary>
        /// Tests proxies for alive and Google passed.
        /// </summary>
        /// <param name="sender"></param>
        /// <param name="e"></param>
        private void btnTestProxies_Click(object sender, EventArgs e)
        {
            proxystested = 0;
            proxySearchPassed = 0;
            proxyTested();
            searchPassedProxy();

            this.proxyTestArray = new BackgroundWorker[this.proxyThreads];

            pr = new Structures.ProxyStruct[this.proxyThreads];

            for (int j = 0; j < this.proxyThreads; j++)
            {
                pr[j].setIP(listViewProxy.Items[j].SubItems[1].Text);
                pr[j].setPort(Convert.ToInt32(listViewProxy.Items[j].SubItems[2].Text));
                pr[j].setLanguage(this.language);
                pr[j].setProxyNo(j);
                pr[j].setHttps(chkTestProxyHttps.Checked);
                pr[j].setSocketSendTimeout(this.proxySendTimeout);
                pr[j].setSocketReceiveTimout(this.proxyReceiveTimeout);
                pr[j].setSearchTimout(this.proxySearchTimeout);

                // Set up array of objects for proxy testings
                proxyTestArray[j] = new BackgroundWorker();
                proxyTestArray[j].DoWork += new DoWorkEventHandler(aliveTest_DoWork);
                proxyTestArray[j].RunWorkerCompleted += new RunWorkerCompletedEventHandler(aliveTest_RunWorkerCompleted);
                proxyTestArray[j].ProgressChanged += new ProgressChangedEventHandler(aliveTest_ProgressChanged);
                proxyTestArray[j].WorkerReportsProgress = true;
                proxyTestArray[j].WorkerSupportsCancellation = true;
                proxyTestArray[j].RunWorkerAsync(pr[j]);
            }
        }

At the other end, you can set a proxy for the object like this:
Code:
                    if (count != 0)
                    {
                        IP = this.listViewProxy.Items[index].SubItems[1].Text;
                        Port = Convert.ToInt32(this.listViewProxy.Items[index].SubItems[2].Text);
                        HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(requestUriString);
                        WebProxy proxy = new WebProxy(IP, Port);
                        webRequest.Timeout = this.proxySearchTimeout;
                        webRequest.Proxy = proxy;
                        webRequest.UserAgent = usrAgent;
                    }
 
Last edited:
Sorry to bump this old thread, but I didn't want to make a new one, because I have the exact same question.

The above codes work really well, so now the next step. How to incorporate rotating proxies into this?

Would you use some sort of refresh method with random proxy assignment?


Have a list of proxies
Lock the list and take a proxy in sequence

From memory
Code:
//your list of proxies
List<Webproxy>listProxies = loadProxies();

//used to lock so only one thread can access the list at any one time
var objLock = new object();
//used to grab the next proxy in sequence
var cntrProxy = 0;

//simple threaded parallel foreach
Parallel.Foreach(listAccounts,parallelOpts,(acc)=>{
  var proxy; 
  lock(objLock){
    if(cntrProxy>=listProxies.Count) cntrProxy=0;
    proxy=listProxies[cntrProxy];
   cntrProxy++;
  }

 //do what you want with proxy and acc

});
 
Sorry to bump this old thread, but I didn't want to make a new one, because I have the exact same question.

The above codes work really well, so now the next step. How to incorporate rotating proxies into this?

Would you use some sort of refresh method with random proxy assignment?

Each time you trigger the request you assign it a random proxy. These proxies can be selected from a list or you can use a constantly tested/updated database of proxies. You can also add support for tracking how many connections are currently being used per proxy to act as a rate limiter to avoid A: Crashing the proxy B: getting it banned by the service you're connecting to.


But yah, already some examples posted for you, just wanted to add my 2c
 
Back
Top