Question about scraping a mobile app

[blackDobe]

I need help with fetching an authorization token from a mobile app.
I used Charles proxy to read how an authorization token is fetched from my target app.
I set the headers and data exactly the same as the mobile app does with a python script using curl_cffi and a smartproxy proxy.
And I get 400 and cloudflare is in the content of the response.

How are they detecting my requests?

 

[followerise]

They detect you because they see, that your request is not from the real browser (they see you cant run js code). If you need to bypass Cloudflare you would need to use headerless browser library, like selenium.

 

[blackDobe]

They detect you because they see, that your request is not from the real browser (they see you cant run js code). If you need to bypass Cloudflare you would need to use headerless browser library, like selenium.

This is not a web app, but a mobile one so I can't use selenium...

Thanks for the reply though

 

[SUPREME.LEADER]

Use auto scraping chrome extension.

 

[blackDobe]

Use auto scraping chrome extension.

This is a mobile app that does api calls and I want to fake the mobile app when I'm making requests to their api

 

[The Undefined]

By exactly, how exactly are we talking about?

E.g. you may pass the data using Binary, GraphQL, Raw (JSON, etc.), x-www-form-urlencoded, form-data. Each of them have different header that is being sent to the server.
Or the method is off (E.g. PUT vs PATCH), etc.

 

[ZeeDigital]

The server might employ anti-bot measures, such as Fail2Ban. I suggest you read the documentation on that since majority of servers run on linux.

 

[blackDobe]

By exactly, how exactly are we talking about?

E.g. you may pass the data using Binary, GraphQL, Raw (JSON, etc.), x-www-form-urlencoded, form-data. Each of them have different header that is being sent to the server.
Or the method is off (E.g. PUT vs PATCH), etc.

I'm using curl_cffi for the requests. I think that library is the best for these kind of operations. All the headers are the same...
The weird thing is I tried the same request with Postman and it worked, with the same exact body and headers

 

[blackDobe]

The server might employ anti-bot measures, such as Fail2Ban. I suggest you read the documentation on that since majority of servers run on linux.

I use proxies and I rotate them all the time. Also I think they use Cloudflare for that

 

[Msuin]

If you use some residential proxies and rotate the IP’s , can work for you

 

[truehero]

I use proxies and I rotate them all the time. Also I think they use Cloudflare for that

That can be issue, because of proxy. 2 possible issues:
- Cloudflare detects that is proxy, by some IPs database ranges
- in cloudflare can be configured different restrictions for different countries. For example, for ips from your country, cloudflare will buypass your request without strict checking, when you make request from another country (that is not the app auditory location) you will be checked more strict by cloudflare

First, just try to get a clear proxy from your current country and test requests.

 

[blackDobe]

If you use some residential proxies and rotate the IP’s , can work for you

Yeah, I already do that with smartproxy.com. But the app really increased their security in the last few days

 

[blackDobe]

That can be issue, because of proxy. 2 possible issues:
- Cloudflare detects that is proxy, by some IPs database ranges
- in cloudflare can be configured different restrictions for different countries. For example, for ips from your country, cloudflare will buypass your request without strict checking, when you make request from another country (that is not the app auditory location) you will be checked more strict by cloudflare

First, just try to get a clear proxy from your current country and test requests.

This is 100% true, but using the same proxy and the same request(method, headers, body) works with Postman and doesn't work on my python script using curl_cffi for the requests

 

[ReverseEngineering]

How are they detecting my requests?

Did you check that there are no hidden requests? e. g. via HTTP/2.0 aka SPDY or HTTP/3.0 aka QUIC

Not all requests are visible at Charles, you should have a look at mitmproxy also.

 

[truehero]

This is 100% true, but using the same proxy and the same request(method, headers, body) works with Postman and doesn't work on my python script using curl_cffi for the requests

So, then it’s impossible, that’s they are equal, when one is working, second - not. There are 100% some differences: different user agents (often libs add own user agent by defaul, like postman, or pythonbot). Can you send your code?

Or you can create a web page, that prints all info about request (cookie, headers, body). And first send from your script, then from postman. And compare them, what difference. I believe, there is a difference