Question about scraping a mobile app

B

blackDobe

Newbie
Joined
Dec 14, 2023
Messages
7
Reaction score
0
I need help with fetching an authorization token from a mobile app.
I used Charles proxy to read how an authorization token is fetched from my target app.
I set the headers and data exactly the same as the mobile app does with a python script using curl_cffi and a smartproxy proxy.
And I get 400 and cloudflare is in the content of the response.

How are they detecting my requests?
 
Advertise on BHW
They detect you because they see, that your request is not from the real browser (they see you cant run js code). If you need to bypass Cloudflare you would need to use headerless browser library, like selenium.
 
followerise said:
They detect you because they see, that your request is not from the real browser (they see you cant run js code). If you need to bypass Cloudflare you would need to use headerless browser library, like selenium.
Click to expand...
This is not a web app, but a mobile one so I can't use selenium...

Thanks for the reply though
 
Use auto scraping chrome extension.
 
By exactly, how exactly are we talking about?

E.g. you may pass the data using Binary, GraphQL, Raw (JSON, etc.), x-www-form-urlencoded, form-data. Each of them have different header that is being sent to the server.
Or the method is off (E.g. PUT vs PATCH), etc.
 
The server might employ anti-bot measures, such as Fail2Ban. I suggest you read the documentation on that since majority of servers run on linux.
 
The Undefined said:
By exactly, how exactly are we talking about?

E.g. you may pass the data using Binary, GraphQL, Raw (JSON, etc.), x-www-form-urlencoded, form-data. Each of them have different header that is being sent to the server.
Or the method is off (E.g. PUT vs PATCH), etc.
Click to expand...
I'm using curl_cffi for the requests. I think that library is the best for these kind of operations. All the headers are the same...
The weird thing is I tried the same request with Postman and it worked, with the same exact body and headers
 
ZeeDigital said:
The server might employ anti-bot measures, such as Fail2Ban. I suggest you read the documentation on that since majority of servers run on linux.
Click to expand...
I use proxies and I rotate them all the time. Also I think they use Cloudflare for that
 
If you use some residential proxies and rotate the IP’s , can work for you
 
blackDobe said:
I use proxies and I rotate them all the time. Also I think they use Cloudflare for that
Click to expand...
That can be issue, because of proxy. 2 possible issues:
- Cloudflare detects that is proxy, by some IPs database ranges
- in cloudflare can be configured different restrictions for different countries. For example, for ips from your country, cloudflare will buypass your request without strict checking, when you make request from another country (that is not the app auditory location) you will be checked more strict by cloudflare

First, just try to get a clear proxy from your current country and test requests.
 
Msuin said:
If you use some residential proxies and rotate the IP’s , can work for you
Click to expand...
Yeah, I already do that with smartproxy.com. But the app really increased their security in the last few days
 
truehero said:
That can be issue, because of proxy. 2 possible issues:
- Cloudflare detects that is proxy, by some IPs database ranges
- in cloudflare can be configured different restrictions for different countries. For example, for ips from your country, cloudflare will buypass your request without strict checking, when you make request from another country (that is not the app auditory location) you will be checked more strict by cloudflare

First, just try to get a clear proxy from your current country and test requests.
Click to expand...
This is 100% true, but using the same proxy and the same request(method, headers, body) works with Postman and doesn't work on my python script using curl_cffi for the requests
 
blackDobe said:
How are they detecting my requests?
Click to expand...

Did you check that there are no hidden requests? e. g. via HTTP/2.0 aka SPDY or HTTP/3.0 aka QUIC

Not all requests are visible at Charles, you should have a look at mitmproxy also.
 
blackDobe said:
This is 100% true, but using the same proxy and the same request(method, headers, body) works with Postman and doesn't work on my python script using curl_cffi for the requests
Click to expand...
So, then it’s impossible, that’s they are equal, when one is working, second - not. There are 100% some differences: different user agents (often libs add own user agent by defaul, like postman, or pythonbot). Can you send your code?

Or you can create a web page, that prints all info about request (cookie, headers, body). And first send from your script, then from postman. And compare them, what difference. I believe, there is a difference
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top