apex1
Regular Member
- May 29, 2015
- 217
- 182
Multiprocessing is kicking my ass 

I've read over 30 tutorials and watched countless videos and still can't write my own custom multi-threaded script in the way I need. I'm on my 3rd day of doing nothing but trying to figure this out lol.
Here's the issue.. I have a script that's very basic. It does the following:
get URL1 and scrape data
get URL1 and scrape data
get URL1 and scrape data
get URL1 and scrape data
get URL1 and scrape data
get URL2 and scrape data
get URL2 and scrape data
get URL2 and scrape data
get URL2 and scrape data
get URL2 and scrape data
It's not taking 1 unique URL per thread
Apparently I need to use queues, locks, pools, manager, and a concurrent future pool executor whatever the f that is. Another problem is say I have an odd number of URLs and I'm running 4 threads, won't that give me an error? Do I have to split my URL list by the amount of threads and assign a number amount to each?
Any programming Gods out there? Bless me with some of your wisdom please
I've read over 30 tutorials and watched countless videos and still can't write my own custom multi-threaded script in the way I need. I'm on my 3rd day of doing nothing but trying to figure this out lol.
Here's the issue.. I have a script that's very basic. It does the following:
- Declares a list (of URLs)
- Creates a loop to go through all URLs
- Scrapes URL source with requests
- Saves results
get URL1 and scrape data
get URL1 and scrape data
get URL1 and scrape data
get URL1 and scrape data
get URL1 and scrape data
get URL2 and scrape data
get URL2 and scrape data
get URL2 and scrape data
get URL2 and scrape data
get URL2 and scrape data
It's not taking 1 unique URL per thread
Apparently I need to use queues, locks, pools, manager, and a concurrent future pool executor whatever the f that is. Another problem is say I have an odd number of URLs and I'm running 4 threads, won't that give me an error? Do I have to split my URL list by the amount of threads and assign a number amount to each?
Any programming Gods out there? Bless me with some of your wisdom please