How do I extract the exact number of an Instagram Profile Post Amount? - (Python - Requests/BeautifulSoup)

[CoderFromHell]

from bs4 import BeautifulSoup
import requests

r = requests.get('https://www.instagram.com/worldstar/?hl=en')
#Link of the Instagram Profile that I would like to Scrape Data from

def parse_data(s):
    
    # creating a dictionary
    data = {}
    
    # splittting the content
    # then taking the first part
    s = s.split("-")[0]
    
    # again splitting the content
    s = s.split(" ")
    
    # assigning the values
    data['Post'] = s[4]
    
    # returning the dictionary
    PostAmount = data['Post']
    return PostAmount

soup = BeautifulSoup(r.text, 'lxml')

meta = soup.find('meta', property="og:description")
#found this from the html source of any instagram profile source code

print (parse_data(meta.attrs['content']))
#prints the description of an instagram profile; their followers,
#following amount and post amount

This code above works only on pages with 9,999 Amount of Post and below. Pages with more than 10,000 Post return 10.1k or 27.k. Im trying to aquire the exact amount of post from an Instagram Profile. Is their another way I can do that?

 

[iwillnbd]

Why not use a 3rd party website to get those stats?

So instead of scraping IG for the post count, you would scrape a website like the one mentioned below which displays the users exact post count.
https://igstats.net/report/joerogan/instagram
This approach, however, would only work for public accounts.

 

[CoderFromHell]

Why not use a 3rd party website to get those stats?

So instead of scraping IG for the post count, you would scrape a website like the one mentioned below which displays the users exact post count.
https://igstats.net/report/joerogan/instagram
This approach, however, would only work for public accounts.

wow...appreciate this. I ended up kinda almost finding a solution shortly after I posted this as well, which involved digging through the javascript that I get back from my get requests, but I'll honestly try this instead and see if it works.

 

[rootjazz]

Pull the details page:
https://www.instagram.com/worldstar/?__a=1
then pull: the value in it's number form

edge_owner_to_timeline_media":{"count":83776,"

 

[CoderFromHell]

Pull the details page:
https://www.instagram.com/worldstar/?__a=1
then pull: the value in it's number form

edge_owner_to_timeline_media":{"count":83776,"

That’s what I was trying. How would one go about printing it? The code I came up with find’s the exact post number, but prints the rest of the JavaScript out for some reason. I’ll post the code.

 

[IYASS]

That’s what I was trying. How would one go about printing it? The code I came up with find’s the exact post number, but prints the rest of the JavaScript out for some reason. I’ll post the code.

You need Python JSON

Check: https://www.w3schools.com/python/python_json.asp

 

[hendovic]

Something like this might work :

import json
import httplib

conn = httplib.HTTPSConnection("www.instagram.com")
conn.request("GET", "/worldstar/?__a=1")

data = conn.getresponse()

json_data = json.loads(data.read())

print json_data["graphql"]["user"]["edge_owner_to_timeline_media"]["count"]

 

[rootjazz]

That’s what I was trying. How would one go about printing it? The code I came up with find’s the exact post number, but prints the rest of the JavaScript out for some reason. I’ll post the code.

As others have said, a json library is probably the best way, or you can probably get away with a hacky regex. But a JSON library would be much easier / robust way to do it

 

[CoderFromHell]

import requests

r = requests.get('https://www.instagram.com/domislivenews/').text


#Gets the followers amount                  FOLLOWERS
start = '"edge_followed_by":{"count":'
end = '},"followed_by_viewer"'
followers= r[r.find(start)+len(start):r.rfind(end)]

#Gets the following amount            FOLLOWING
start = '"edge_follow":{"count":'
end = '},"follows_viewer"'
following= r[r.find(start)+len(start):r.rfind(end)]

#Gets the number of post                 POST
start = '"edge_owner_to_timeline_media":{"count":'
end = ',"page_info":{"has_next_page"'           #ERROR HERE
posts= r[r.find(start)+len(start):r.rfind(end)]

print(posts, followers, following)

Was thinking something along the lines here. Most of this code I gathered here is from stack overflow; Im pulling these strings from the GET response of any Instagram profile. The error is in the end variable.

 

[CoderFromHell]

Something like this might work :

import json
import httplib

conn = httplib.HTTPSConnection("www.instagram.com")
conn.request("GET", "/worldstar/?__a=1")

data = conn.getresponse()

json_data = json.loads(data.read())

print json_data["graphql"]["user"]["edge_owner_to_timeline_media"]["count"]

This works well actually, thank you.

 

[GoGuerilla]

Damn, I'm surprised you can access instagram with pure http requests.

 

[rootjazz]

Damn, I'm surprised you can access instagram with pure http requests.

you can pull about 100-200 profiles within (n) time period without being logged in. After that they will redirect you to the login page unless you change IP

 

[CoderFromHell]

you can pull about 100-200 profiles within (n) time period without being logged in. After that they will redirect you to the login page unless you change IP

Thats exactly what just happened to me and it caught me off guard

 

[rootjazz]

It's a new limit, well not new new, maybe 2 years old now. Really limited the amount of scraping you can do. Either you need to automate the login system (this is difficult as you need to decode the signature routine. Or if you are working with one account, login with your browser, then pull the cookies from a network sniffer like fiddler / charles / burp what ever you use

 

[Standartuye]

It's a new limit, well not new new, maybe 2 years old now. Really limited the amount of scraping you can do. Either you need to automate the login system (this is difficult as you need to decode the signature routine. Or if you are working with one account, login with your browser, then pull the cookies from a network sniffer like fiddler / charles / burp what ever you use

Mate
Is there a software that we can scrape instagram e-mail addresses with 100 different IG accounts with proxies?
I tried to look for it but couldn't find a promising one. Heard about growmeorganic but I am not sure

 

[Standartuye]

Our product can do that. Link to BST. I am not aware of other programs, but it is a simple function, so I would think the other main bots could do it. Jarvee / followliker, but you would need to contact them and check. But if you already have the URLs you want to scrape and the proxies, then all you need is a web page puller and email scraper. I would imagine these type of programs exist, possibly free ones (but I don't know for sure). There is nothing IG specific about pulling aweb page and scraping for email. The only reason you might want an IG bot for this is, if you want to utilize scraping of profiles with filters to then feed into the email scraping

But I think, in order to extract contact email, software needs to send mobile API requests right?

 

[rootjazz]

But I think, in order to extract contact email, software needs to send mobile API requests right?

Business contact email yes, you would need to be on the mobile API. As this thread is about web scrapes, I assumed you meant scraping emails from the bio (as you cannot get contact details from the web page).

But my response above is still valid. Scraping emails from contact details is a simple function and any competent IG bot should be able to do this