Arc717
Newbie
- Aug 7, 2018
- 17
- 10
I am currently web-scraping a bunch of similar pages in Python. Here is my Setup:
- Using Requests sessions to get data, no Selenium or JavaScript involved
- Using Threads to make concurrent requests, not Asyncio or Aiohttp (Aiohttp is too buggy) - 55 Threads
- Maintaining a pool of Request Session Objects and reusing them for requests so I can avoid reopening TCP connections and save scraping time - 60 sessions
- Parsing 5 html elements using regex from each page and storing these elements in a list
- This is just one Python Script, no multiprocessing, and no multiple scripts
I can get through about 10,000 pages this way in roughly 7 minutes when the server I am scraping from is not under load near midnight. Usually the server responds to each request in under half a second. When it is under load though, I may wait multiple seconds for a request, with a decent amount of timeouts.
My question is this: Would I see a large performance boost using C++ to web-scrape with the same amount of current requests (55) at once? Would it justify going through the effort of learning C++?
I am aware of Python's GIL and the fact that interpreted languages are slower than compiled ones. I know that I can just buy a better server and use the multiprocessing module in Python to get more performance out of my script, but I was wondering if recoding in C++ was worth it.
If I can do 10,000 pages in 7 minutes on Python without multiprocessing and with 55 concurrent requests at any one time, how many minutes would it take to do the same amount of pages in C++ with only 55 concurrent requests?
- Using Requests sessions to get data, no Selenium or JavaScript involved
- Using Threads to make concurrent requests, not Asyncio or Aiohttp (Aiohttp is too buggy) - 55 Threads
- Maintaining a pool of Request Session Objects and reusing them for requests so I can avoid reopening TCP connections and save scraping time - 60 sessions
- Parsing 5 html elements using regex from each page and storing these elements in a list
- This is just one Python Script, no multiprocessing, and no multiple scripts
I can get through about 10,000 pages this way in roughly 7 minutes when the server I am scraping from is not under load near midnight. Usually the server responds to each request in under half a second. When it is under load though, I may wait multiple seconds for a request, with a decent amount of timeouts.
My question is this: Would I see a large performance boost using C++ to web-scrape with the same amount of current requests (55) at once? Would it justify going through the effort of learning C++?
I am aware of Python's GIL and the fact that interpreted languages are slower than compiled ones. I know that I can just buy a better server and use the multiprocessing module in Python to get more performance out of my script, but I was wondering if recoding in C++ was worth it.
If I can do 10,000 pages in 7 minutes on Python without multiprocessing and with 55 concurrent requests at any one time, how many minutes would it take to do the same amount of pages in C++ with only 55 concurrent requests?