Looking to learn blackhat scraping methods

[mojalai]

Hello - newb here. I joined because I heard it's a good source for purchasing things like proxies and fake instagram accounts. I've been learning to program and want to build a saas eventually and expect that this place will have a lot of useful resources.

 

[todordonev]

What is your tech stack mate?

 

[TechEinstien]

What are you trying to learn?

 

[mojalai]

What is your tech stack mate?

Django with Celery and Redis for backend and Vanilla JS for frontend.

 

[mojalai]

What are you trying to learn?

Right now I'm trying to figure out how to scrape emails from Instagram. There seem to be products out there that are capable of this, but emails aren't shown in the browser unless a user explicitly puts it in their bio or something. So I'm now wondering if the products that are out there are actually just doing some additional webcrawling of non-Instagram pages to try to get emails corresponding to Instagram users. For that I've just begun digging into the Python library Scrapy. Eventually, if I'm able to figure out how to get the emails, I'll likely need to then figure out the best way(s) to scrape without being banned. I currently only know basic ways such as rotating user agents and random wait times before requests. I also keep seeing people talk about the importance of proxies or VPS's, but I haven't gotten to the point of really researching these topics yet.

 

[todordonev]

Right now I'm trying to figure out how to scrape emails from Instagram. There seem to be products out there that are capable of this, but emails aren't shown in the browser unless a user explicitly puts it in their bio or something. So I'm now wondering if the products that are out there are actually just doing some additional webcrawling of non-Instagram pages to try to get emails corresponding to Instagram users. For that I've just begun digging into the Python library Scrapy. Eventually, if I'm able to figure out how to get the emails, I'll likely need to then figure out the best way(s) to scrape without being banned. I currently only know basic ways such as rotating user agents and random wait times before requests. I also keep seeing people talk about the importance of proxies or VPS's, but I haven't gotten to the point of really researching these topics yet.

What you are looking for is mobile emulation, only then you can see the email, also you do not need to emulate browser just send very specific http requests. You will need to connect your phone with your home router, intercept the requests and you will see the api endpoints. This is not an official API so there is no documentation. Basically its very doable but you will have to figure it out on your own. Ive done it and it works and still works.

 

[mojalai]

Interesting. Thank you for the direction. I wouldn't have been able to think of this on my own.

 

[Conjecting]

First and foremost, welcome to the forum. You'll certainly be able to find information on blackhat scraping methods here, and programming is a highly valuable skill that will always be in demand so I encourage you to stick with it

 

[tesselate]

What you are looking for is mobile emulation, only then you can see the email, also you do not need to emulate browser just send very specific http requests. You will need to connect your phone with your home router, intercept the requests and you will see the api endpoints. This is not an official API so there is no documentation. Basically its very doable but you will have to figure it out on your own. Ive done it and it works and still works.

A couple questions... Would the api requests be encrypted since its a mobile device? I've been foraying into the world of mobile application testing and even after installing the correct certs my mitm proxy still doesn't pick up requests for things like Google, LinkedIn, etc. The applications stop working all together on the mobile device at all. Any idea what I am doing wrong?

 

[mojalai]

What you are looking for is mobile emulation, only then you can see the email, also you do not need to emulate browser just send very specific http requests. You will need to connect your phone with your home router, intercept the requests and you will see the api endpoints. This is not an official API so there is no documentation. Basically its very doable but you will have to figure it out on your own. Ive done it and it works and still works.

So I tried going the route you mentioned. I initially downloaded Wireshark, but I couldn’t quickly tell if the requests could be decoded from the packets by Wireshark. I then tried Burp Suite and was able to get it to intercept traffic on my phone. After logging into Instagram and searching for a user, I came across two interesting paths. The first had “graphql/query/?query_hash=[some long hash with an ‘id’ and ‘first’ variables]” after Instagram dot com. After inputting this path into a browser, a JSON-looking page came up but without anything related to emails. The second path seemed to give more interesting attributes. The path was Instagram dot com /{username}/?__a=1&__d=dis. One of the attributes that came up in the browser is “business_email”. Unfortunately, for the several accounts I tried, none had values for this field. Does it sound like I followed your direction correctly? Or am I missing something? Thanks for your help.

 

[mojalai]

A couple questions... Would the api requests be encrypted since its a mobile device? I've been foraying into the world of mobile application testing and even after installing the correct certs my mitm proxy still doesn't pick up requests for things like Google, LinkedIn, etc. The applications stop working all together on the mobile device at all. Any idea what I am doing wrong?

With Burp Suite, I had to keep pressing "Forward" in order to have the traffic pass through to the destination initially requested by the mobile's browser. Maybe there's similar functionality with whatever you're using? Also, maybe I'm not understanding you correctly as I'm new to Burp Suite and a lot of this stuff.

 

[todordonev]

So I tried going the route you mentioned. I initially downloaded Wireshark, but I couldn’t quickly tell if the requests could be decoded from the packets by Wireshark. I then tried Burp Suite and was able to get it to intercept traffic on my phone. After logging into Instagram and searching for a user, I came across two interesting paths. The first had “graphql/query/?query_hash=[some long hash with an ‘id’ and ‘first’ variables]” after Instagram dot com. After inputting this path into a browser, a JSON-looking page came up but without anything related to emails. The second path seemed to give more interesting attributes. The path was Instagram dot com /{username}/?__a=1&__d=dis. One of the attributes that came up in the browser is “business_email”. Unfortunately, for the several accounts I tried, none had values for this field. Does it sound like I followed your direction correctly? Or am I missing something? Thanks for your help.

Try on business accounts, for example accounts that come up after searching for seo agency or something like that.