Python Scraper is slow as hell

[madblackhatter1000]

I'm trying to build an analytics social graph tool that shows who's connected to who.

I'm only going 2 levels deep - scraping the followers of followers.

Example: if Main account has 1000 followers, and each follower has 100-500 followers = 100,000-500,000 in total scraped list.

I made the python scraper that easily scraped the first level (1000 followers), but seems to be getting 1-5 minute temporary blocks and throttled with each additional Level 2 follower-of-a-follower. After 12-Hours, its still only scraped 50,000 of the total 100K-500K follower-of-a-follower.

Any way to make this go faster?

BTW, I'm a newbie Python Developer, so I'm pretty sure I made 100 mistakes in my code. I tested my script successfully a few times on small accounts with low followers with low followers (a total of 500 in all level 1 + level 2 combined).

 

[CoolNicks]

What platform you are talking about?
What library/framework you are using to scarp?
Do you use multithreading/multiprocessing ?
What does the temporary block you are refering to? Does the site imposing rate limiting ?
Are you scraping from same ip?
Are you logged into the site?

Hope you got the answers.

 

[Stem]

Where are scraping from? Twitter? It might not be related to script but to the source.

 

[The Undefined]

As the thread is the Instagram section, He's probably referring to the Instagram platform.
I'm not an expert in Instagram automation but, by the tests, I've made on their private (mobile) APIs and public (web) APIs these are some notes to increase your scraping speed:

1. Try with multiple IP addresses
   Run 5 tor instances (with control port enabled, reset tor circuit for every 2 or 3 average users you scraped, or per level if your script is written in recursive format *)
2. To decrease the tor circuit rests I'd suggest logging into one account per instance and fake that you're using a dynamic VPN (which is poor in term of connection and resets every N minutes) by this you're providing yourself a way to increase API calls.
3. Be aware of the browser details which is sent by your crawler, if you've logged into an account per instance as I said above, make sure you change details every ~500 requests if you didn't log into any account change it every 20-30 requests.
4. If you've developed your script based on the web platform, Make sure to get user page before calling the follower's list API.

Best regards

 

[madblackhatter1000]

As the thread is the Instagram section, He's probably referring to the Instagram platform.
I'm not an expert in Instagram automation but, by the tests, I've made on their private (mobile) APIs and public (web) APIs these are some notes to increase your scraping speed:

1. Try with multiple IP addresses
   Run 5 tor instances (with control port enabled, reset tor circuit for every 2 or 3 average users you scraped, or per level if your script is written in recursive format *)
2. To decrease the tor circuit rests I'd suggest logging into one account per instance and fake that you're using a dynamic VPN (which is poor in term of connection and resets every N minutes) by this you're providing yourself a way to increase API calls.
3. Be aware of the browser details which is sent by your crawler, if you've logged into an account per instance as I said above, make sure you change details every ~500 requests if you didn't log into any account change it every 20-30 requests.
4. If you've developed your script based on the web platform, Make sure to get user page before calling the follower's list API.

Best regards

Thank You everyone. Yes, this is for INSTAGRAM. At the moment just testing on 1 residential ip (home) which the ISP rotates the IP-Address every 3-15 minutes. Just a single thread using the API JSON calls as listed on the Instagram Dev website.

Great idea about rotating the user-agent switching, and adding multiple instances

I had to export my cookies/token and embed them in the Python script to get this to work.

 

[madblackhatter1000]

Wow 60-Hours to python scrape 2000 Followers Followings (the people my followers follow). I haven't implemented the multithreading/proxies/user-agent switching yet into the script.

Are there any existing IG python scripts out there in the public that I should check out and learn from (for making multi-threading/user-agent switching,etc....)

 

[naughtynewell]

60 hours to do that is... Awful.

Given that you're using selenium:

- Every time you call

webdriver.Chrome

or similar the computer has to start a new instance of Chrome. So minimise the number of times you do that if you can.

- Close the browser once you've finished with it. Make sure that you have code to cost and quit the browser instance in your bot, otherwise all the Browser instances from previous runs will sit there hogging resources until your computer shits the bed

 

[iwillnbd]

Wow 60-Hours to python scrape 2000 Followers Followings (the people my followers follow). I haven't implemented the multithreading/proxies/user-agent switching yet into the script.

Are there any existing IG python scripts out there in the public that I should check out and learn from (for making multi-threading/user-agent switching,etc....)

InstaPY is an open-source Instagram bot that's built using selenium on python.
Without multiple threads, proxies etc. your bot will obviously be slow...

 

[madblackhatter1000]

I was actually using the direct api calls (I think they're called JSON?) mentioned in the IG Development Site.

Can I scrape more data, and scrape it faster using selenium/chrome versus the direct json calls?

 

[RobertTheShark]

Are there any existing IG python scripts out there in the public that I should check out and learn from (for making multi-threading/user-agent switching,etc....)

A while ago someone here shared a python based IG bot for the raspi, you should check that out. It was in the "Members Download" section I believe.

I was actually using the direct api calls (I think they're called JSON?) mentioned in the IG Development Site.

Can I scrape more data, and scrape it faster using selenium/chrome versus the direct json calls?

Yes, JSON (its a notiation form) is the standard way of sharing data between different applications (API -> your .py script).
In general what might be the reason that your bot is slow is the IG api. One way could overcome that is to use different Browser Agents + Proxys + different IG accounts for the scrapping part. IG API may slow down if the see someone is crawling massive data from them.

Or you could directly forget about the IG api (I have never worked with that) and just go on use a browser engine / framework (like selenium) to scrape the followers on that way (maybe thats gonna be faster, if everything is implemented right). Also make sure to again, use different IG accounts, proxys etc.

You have always to think that way: IG has the ressources to log and track everything down. They instantly know if a bot is on their site, but like Google or any other bigger site, they let the bot to do its work but after some time they do slow things down or shut it down completly (banned etc.). So try to make your bot act as legit as possible.


Some other thoughts:
1. Python should not be the problem here (in terms of performance, although any low-level language like C would be faster, however development time would go brutally up), so stay with python.

2. Multithreading would definetly improve your bot. But I would not focus on that from the beginning, implement the part with browser agents and proxys first. Thats more important.

 

[The Undefined]

Hey there,
You can always scrape more but, did you just said you're using the official public APIs (mentioned on instagram.com/developer)?!
If you're doing this you're wasting your money and energy, take a look at application level APIs; for example, the APIs which are working to provide the user experience for the main website (It's a react based app when you log in which can simply be reverse-engineered for your bot).
For your question, It all depends on you and how you implement it so, the answer would be yes and no.
By the way, according to the JSON website, It's a standard for data interchange and it's the short form of JavaScript Object Notation. You can read more about it by reading ECMA-404 standard.

 

[Alex D.]

I was actually using the direct api calls (I think they're called JSON?) mentioned in the IG Development Site.

Can I scrape more data, and scrape it faster using selenium/chrome versus the direct json calls?

Don't use selenium it will only slow you even more.

 

[Marcel Freiberg]

Disclaimer has been a few month the last time I scraped followers of followers but I were able to scrape about a few 100ks of followers per 3 minutes with one account/server.

Used selenium to and a headless chrome browser and simulated client calls to the graph api.

I would suggest a 2step process. A logged in step to scrape simple list of followers and a anonymous / multithreaded (keyword celery behind tor rotating proxy) to receive details.