How to make Python multi-threaded?

IAmDaze · Jan 28, 2019

I am just learning a Python. I have made IP scraping bot and it saved output as IP

ORT in text time. Now my next step is to check if the scraped IPs are alive or not. I have already done it but its slow since pinging take some time. So I want to make it multi-threaded so that it can check multiple IPs at once.

Code:

import os

readfile = open('test.txt', 'r').read().split('\n')
for ip in readfile :
    respose = os.system('ping -n 1 '+ip[0:ip.find(':')])
    if respose == 0 :
        print(ip + ' is up!')
    else :
        print(ip + ' is down!')

G4143 · Jan 31, 2019

Why make it multi-threading? I would explore asynchronous(single threaded) functionality since your requests are blocking. Just make a bunch of asynchronous(single threaded) calls and they should response when they time-out or have a something valid to return.

I'm not a Python programmer but I would start here if I was:

https://docs.python.org/3/library/asyncio.html

http://masnun.rocks/2016/10/06/async-python-the-different-forms-of-concurrency/

Also note: Python's multi-threading(where you spawn multi threads) is butt ass slow.

IAmDaze · Feb 1, 2019

G4143 said:
Why make it multi-threading? I would explore asynchronous(single threaded) functionality since your requests are blocking. Just make a bunch of asynchronous(single threaded) calls and they should response when they time-out or have a something valid to return.

I'm not a Python programmer but I would start here if I was:

https://docs.python.org/3/library/asyncio.html

http://masnun.rocks/2016/10/06/async-python-the-different-forms-of-concurrency/

Also note: Python's multi-threading(where you spawn multi threads) is butt ass slow.

That is exactly what I was looking for. Thank you so much!

Miraz0 · Feb 6, 2019

This is the code I used to make my bot multithreaded:

(It might not be 100% right, had to cut out some parts.)
But it's the base of it.

Code:

import threading

def main(threadID, config):
    print ("Thread [%s]: Starting..." % (threadID))
    #do stuff here.


if __name__ == '__main__':
    maxThread = int(input("Amount of threads: "))
    for i in range(maxThread):
        t = threading.Thread(target=threadDef, args=(i, config))
        t.start()

Deen0229 · Apr 2, 2019

If you're doing anything that requires multithreading, I'd learn Golang. It's very easy to learn so it will be fast to start building projects with (within 2-4 weeks) and you'll have a lot more performance. It has something called Goroutines which makes implementing multithreading effortless. (Python can never be truly multithreaded)
It's used a lot in big data, scraping, web app projects. Even Improbable.io use it for SpatialOS distributed game server.

PIEGE · Aug 22, 2019

Its worth mentioning that 'multi-threading' is limited in python.. due to a feature called the GIL: https://wiki.python.org/moin/GlobalInterpreterLock

as others have mentioned, if you want ' concurrency ' in python then making asynchronous calls will work.

anybody claiming that you should switch languages just to achieve high speed concurrency could stand to do a little more research. python is likely plenty fast and capable for what you want and mad easy to learn. when i want concurrency at high speed, i turn to packages that use uvloop ( or just use uvloop directly, may require you to practice / experiment a little )

UV LOOP - blazing fast python networking -> https://magic.io/blog/uvloop-blazing-fast-python-networking/

good luck

Code Docta · Oct 7, 2019

With Python it really depends on what you're doing and it has a solution for everything. If you're doing a CPU bound task like working with images/videos then you'll want to use multiprocessing. If you're doing IO(reading/writing/internet) tasks then multithreading.

The concurrent.futures module for someone new to Python is easier to learn than asyncio and libraries using it. This video explains it nicely.

Proxy_prinz · Dec 4, 2023

It is necessary to separate the concepts:

Asynchrony
Multiprocessing
Multithreading

Before performing the task, I think it’s worth reading about these possibilities, because in each of them, interaction with the computer occurs differently and each tool has its own tasks

Well, that is, you can say this: It’s pointless to go to the store for groceries on a spaceship)

I hope the idea is clear and my message gave new thoughts

How to make Python multi-threaded?

IAmDaze

Junior Member

G4143

Registered Member

IAmDaze

Junior Member

Miraz0

Junior Member

Deen0229

Banned - selling outside of the marketplace.

PIEGE

Newbie

Code Docta

Registered Member

Proxy_prinz

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World