InstaOmega
Regular Member
- Aug 1, 2019
- 264
- 80
Hi BHW community,
I wanted to make this post to educate anyone in the community about how to get started scraping Instagram. I am a big believer in democratizing data and a major component of data aggregation is scraping!
Web scraping is not a new concept and there are many ways to get it done.
One way is to load up the webpage and scrape the html using libraries like Selenium or you can request data from internal urls that IG uses to deliver information to their users.
Here is how to use the internal urls and in my opinion is the easiest way right now to start scraping IG at scale. All you need to do is to take any IG url like https://www.instagram.com/instagram/ and add this to ending "?__a=1".
So your final url would look like this: https://www.instagram.com/instagram/?__a=1
If done properly, you should be seeing a JSON response with the data that IG delivers.
So that is basically the foundation of scraping IG. There is a lot of things you will still need to do to scale the process. For example, some information is only available if you are logged into an IG account. To get around that you will need to login into an IG account before scraping.
Also, you will still need to get around the IP bans so you will need proxies to do that. I recommend mobile proxies but residential do work as well. The IG accounts that you use to scrape may be burned faster with residential proxies.
Overall, scraping Instagram at the bare minimum is pretty easy to do. The hard part is scaling your scraper to be able to scrape millions of Instagram accounts!
Anyways, I hope this helped. Feel free to reach out with any questions. Always happy to help the BHW community!
I wanted to make this post to educate anyone in the community about how to get started scraping Instagram. I am a big believer in democratizing data and a major component of data aggregation is scraping!
Web scraping is not a new concept and there are many ways to get it done.
One way is to load up the webpage and scrape the html using libraries like Selenium or you can request data from internal urls that IG uses to deliver information to their users.
Here is how to use the internal urls and in my opinion is the easiest way right now to start scraping IG at scale. All you need to do is to take any IG url like https://www.instagram.com/instagram/ and add this to ending "?__a=1".
So your final url would look like this: https://www.instagram.com/instagram/?__a=1
If done properly, you should be seeing a JSON response with the data that IG delivers.
So that is basically the foundation of scraping IG. There is a lot of things you will still need to do to scale the process. For example, some information is only available if you are logged into an IG account. To get around that you will need to login into an IG account before scraping.
Also, you will still need to get around the IP bans so you will need proxies to do that. I recommend mobile proxies but residential do work as well. The IG accounts that you use to scrape may be burned faster with residential proxies.
Overall, scraping Instagram at the bare minimum is pretty easy to do. The hard part is scaling your scraper to be able to scrape millions of Instagram accounts!
Anyways, I hope this helped. Feel free to reach out with any questions. Always happy to help the BHW community!
