How do I extract the exact number of an Instagram Profile Post Amount? - (Python - Requests/BeautifulSoup)

CoderFromHell · Jul 29, 2020

Python:

from bs4 import BeautifulSoup
import requests

r = requests.get('https://www.instagram.com/worldstar/?hl=en')
#Link of the Instagram Profile that I would like to Scrape Data from

def parse_data(s):
    
    # creating a dictionary
    data = {}
    
    # splittting the content
    # then taking the first part
    s = s.split("-")[0]
    
    # again splitting the content
    s = s.split(" ")
    
    # assigning the values
    data['Post'] = s[4]
    
    # returning the dictionary
    PostAmount = data['Post']
    return PostAmount

soup = BeautifulSoup(r.text, 'lxml')

meta = soup.find('meta', property="og:description")
#found this from the html source of any instagram profile source code

print (parse_data(meta.attrs['content']))
#prints the description of an instagram profile; their followers,
#following amount and post amount

This code above works only on pages with 9,999 Amount of Post and below. Pages with more than 10,000 Post return 10.1k or 27.k. Im trying to aquire the exact amount of post from an Instagram Profile. Is their another way I can do that?

iwillnbd · Jul 29, 2020

Why not use a 3rd party website to get those stats?

So instead of scraping IG for the post count, you would scrape a website like the one mentioned below which displays the users exact post count.
https://igstats.net/report/joerogan/instagram
This approach, however, would only work for public accounts.

CoderFromHell · Jul 29, 2020

iwillnbd said:
Why not use a 3rd party website to get those stats?

So instead of scraping IG for the post count, you would scrape a website like the one mentioned below which displays the users exact post count.
https://igstats.net/report/joerogan/instagram
This approach, however, would only work for public accounts.

wow...appreciate this. I ended up kinda almost finding a solution shortly after I posted this as well, which involved digging through the javascript that I get back from my get requests, but I'll honestly try this instead and see if it works.

rootjazz · Jul 29, 2020

Pull the details page:
https://www.instagram.com/worldstar/?__a=1
then pull: the value in it's number form

Code:

edge_owner_to_timeline_media":{"count":83776,"

CoderFromHell · Jul 29, 2020

rootjazz said:
Pull the details page:
https://www.instagram.com/worldstar/?__a=1
then pull: the value in it's number form

Code:

edge_owner_to_timeline_media":{"count":83776,"

That’s what I was trying. How would one go about printing it? The code I came up with find’s the exact post number, but prints the rest of the JavaScript out for some reason. I’ll post the code.

IYASS · Jul 29, 2020

CoderFromHell said:
That’s what I was trying. How would one go about printing it? The code I came up with find’s the exact post number, but prints the rest of the JavaScript out for some reason. I’ll post the code.

You need Python JSON

Check: https://www.w3schools.com/python/python_json.asp

hendovic · Jul 29, 2020

Something like this might work :

Code:

import json
import httplib

conn = httplib.HTTPSConnection("www.instagram.com")
conn.request("GET", "/worldstar/?__a=1")

data = conn.getresponse()

json_data = json.loads(data.read())

print json_data["graphql"]["user"]["edge_owner_to_timeline_media"]["count"]

rootjazz · Jul 29, 2020

CoderFromHell said:
That’s what I was trying. How would one go about printing it? The code I came up with find’s the exact post number, but prints the rest of the JavaScript out for some reason. I’ll post the code.

As others have said, a json library is probably the best way, or you can probably get away with a hacky regex. But a JSON library would be much easier / robust way to do it

CoderFromHell · Jul 29, 2020

Python:

import requests

r = requests.get('https://www.instagram.com/domislivenews/').text


#Gets the followers amount                  FOLLOWERS
start = '"edge_followed_by":{"count":'
end = '},"followed_by_viewer"'
followers= r[r.find(start)+len(start):r.rfind(end)]

#Gets the following amount            FOLLOWING
start = '"edge_follow":{"count":'
end = '},"follows_viewer"'
following= r[r.find(start)+len(start):r.rfind(end)]

#Gets the number of post                 POST
start = '"edge_owner_to_timeline_media":{"count":'
end = ',"page_info":{"has_next_page"'           #ERROR HERE
posts= r[r.find(start)+len(start):r.rfind(end)]

print(posts, followers, following)

Was thinking something along the lines here. Most of this code I gathered here is from stack overflow; Im pulling these strings from the GET response of any Instagram profile. The error is in the end variable.

CoderFromHell · Jul 29, 2020

hendovic said:

Something like this might work :

Code:

import json
import httplib

conn = httplib.HTTPSConnection("www.instagram.com")
conn.request("GET", "/worldstar/?__a=1")

data = conn.getresponse()

json_data = json.loads(data.read())

print json_data["graphql"]["user"]["edge_owner_to_timeline_media"]["count"]

This works well actually, thank you.

GoGuerilla · Jul 30, 2020

Damn, I'm surprised you can access instagram with pure http requests.

rootjazz · Jul 30, 2020

GoGuerilla said:
Damn, I'm surprised you can access instagram with pure http requests.

you can pull about 100-200 profiles within (n) time period without being logged in. After that they will redirect you to the login page unless you change IP

CoderFromHell · Jul 30, 2020

rootjazz said:
you can pull about 100-200 profiles within (n) time period without being logged in. After that they will redirect you to the login page unless you change IP

Thats exactly what just happened to me and it caught me off guard

rootjazz · Jul 30, 2020

It's a new limit, well not new new, maybe 2 years old now. Really limited the amount of scraping you can do. Either you need to automate the login system (this is difficult as you need to decode the signature routine. Or if you are working with one account, login with your browser, then pull the cookies from a network sniffer like fiddler / charles / burp what ever you use

Standartuye · Jul 30, 2020

rootjazz said:
It's a new limit, well not new new, maybe 2 years old now. Really limited the amount of scraping you can do. Either you need to automate the login system (this is difficult as you need to decode the signature routine. Or if you are working with one account, login with your browser, then pull the cookies from a network sniffer like fiddler / charles / burp what ever you use

Mate
Is there a software that we can scrape instagram e-mail addresses with 100 different IG accounts with proxies?
I tried to look for it but couldn't find a promising one. Heard about growmeorganic but I am not sure

Standartuye · Jul 30, 2020

rootjazz said:
Our product can do that. Link to BST. I am not aware of other programs, but it is a simple function, so I would think the other main bots could do it. Jarvee / followliker, but you would need to contact them and check. But if you already have the URLs you want to scrape and the proxies, then all you need is a web page puller and email scraper. I would imagine these type of programs exist, possibly free ones (but I don't know for sure). There is nothing IG specific about pulling aweb page and scraping for email. The only reason you might want an IG bot for this is, if you want to utilize scraping of profiles with filters to then feed into the email scraping

But I think, in order to extract contact email, software needs to send mobile API requests right?

rootjazz · Jul 30, 2020

Standartuye said:
But I think, in order to extract contact email, software needs to send mobile API requests right?

Business contact email yes, you would need to be on the mobile API. As this thread is about web scrapes, I assumed you meant scraping emails from the bio (as you cannot get contact details from the web page).

But my response above is still valid. Scraping emails from contact details is a simple function and any competent IG bot should be able to do this

How do I extract the exact number of an Instagram Profile Post Amount? - (Python - Requests/BeautifulSoup)

CoderFromHell

Power Member

iwillnbd

Power Member

CoderFromHell

Power Member

rootjazz

Elite Member

CoderFromHell

Power Member

IYASS

Regular Member

hendovic

Newbie

rootjazz

Elite Member

CoderFromHell

Power Member

CoderFromHell

Power Member

GoGuerilla

Power Member

rootjazz

Elite Member

CoderFromHell

Power Member

rootjazz

Elite Member

Standartuye

Regular Member

Standartuye

Regular Member

rootjazz

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World