Scrapping TikTok with mobile API (search with filters)

[oatsactuary]

There are ways to scrap videos and stats from TikTok for analysis and trends. Most popular libraries use the same endpoints a web browser would use, where you don't require having any type of authentication. Other libraries relay on the mobile endpoints relay on the /aweme endpoints if you have an iid, this is the same you would execute from your phone (which you can use without authentication). I got both working, you can also see the code in libraries like yt-dlp: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/tiktok.py

But I want to take it a step further. What would be really powerful is to have a feed of information coming from a specific hashtag or search, in the last 24 hours, sorted by popularity. This is what we usually do on our phones and this can be used without authentication as well.

Analysing the Android app, we have found the endpoint which is `/aweme/v1/general/search/stream`, but is not as easy I initially thought. TikTok sends parameters in a signed header like:

-H 'X-Argus: DwxHKQeBPWvG....19M/ig=' \
-H 'X-Gorgon: 8404b05e5000...6372af' \

... which makes it difficult to decipher or to know which parameters or how have to be sent. But if we just do a curl, we do get the results from our terminal, meaning that if we are able to workout the X-Argus/X-Gorgon headers, we could do this from an internal dashboard.

Has anyone explored this or cracked the arguments?

Thanks in advance!

 

[DropVS]

Any updates?

 

[Sanjuwa]

Is that header exist in web version too?

 

[DropVS]

Is that header exist in web version too?

yes

 

[RaeAly]

You can find old leaked versions of the signature algorithms on Github if all you need is basic scraping and not anything more complicated than that.

Anything else, you'll need latest algorithms which cost a lot, so if you have any issues with the leaked ones then I'd just recommend using a scraping API from RapidAPI or TikHub.

Be careful with Github btw, a LOT of scammers promising to sell algorithms for cheap when they cost multiple thousands, or just plain malware in the code.

 

[Sanjuwa]

yes

then you can find javascript functions that generate those params

 

[oatsactuary]

I'm looking into using the search filters, which are only available on app... so I guess it's gonna be impossible as I don't know the arguments to be send or encoded

 

[Lukmat]

There are ways to scrap videos and stats from TikTok for analysis and trends. Most popular libraries use the same endpoints a web browser would use, where you don't require having any type of authentication. Other libraries relay on the mobile endpoints relay on the /aweme endpoints if you have an iid, this is the same you would execute from your phone (which you can use without authentication). I got both working, you can also see the code in libraries like yt-dlp: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/tiktok.py

But I want to take it a step further. What would be really powerful is to have a feed of information coming from a specific hashtag or search, in the last 24 hours, sorted by popularity. This is what we usually do on our phones and this can be used without authentication as well.

Analysing the Android app, we have found the endpoint which is `/aweme/v1/general/search/stream`, but is not as easy I initially thought. TikTok sends parameters in a signed header like:

-H 'X-Argus: DwxHKQeBPWvG....19M/ig=' \
-H 'X-Gorgon: 8404b05e5000...6372af' \

... which makes it difficult to decipher or to know which parameters or how have to be sent. But if we just do a curl, we do get the results from our terminal, meaning that if we are able to workout the X-Argus/X-Gorgon headers, we could do this from an internal dashboard.

Has anyone explored this or cracked the arguments?

Thanks in advance!

I was studying x-argus, x-gorgon, x-khronos and probably they added one more
There are a lot of people on github claiming they have for sell, but always "Write me on Telegram" xD
Why they not sell on BHW or something like that, where they can be fast banned?

I was asking some times several models like chatgpt, claude, deepsek and this is one of popular scams.
These keys are not broken, if they would, TikTok would change entire system very fast.


GPT Claude

The TikTok API uses several security headers for request verification and encryption:

[LIST=1]
[*]X-Gorgon: A 84-bit encryption mechanism that helps verify legitimate requests. It's generated using device information, URL parameters, and other request data.
[*]X-Khronos: A timestamp value (not an encryption) used to prevent replay attacks by ensuring requests are processed within a valid time window.
[*]X-Argus: A 164-bit encryption mechanism that evolved from X-Gorgon with more robust security. It incorporates additional parameters and provides stronger protection against unauthorized API access.
[/LIST]
These encryption methods are part of TikTok's API security layer to prevent unauthorized access and scraping. The specific implementation details may have evolved since my knowledge cutoff in October 2024, so there could be updates to the bit lengths or encryption approaches.

Retry
L
are they broken?
Edit

As of my last knowledge update in October 2024, TikTok's security headers (X-Gorgon, X-Argus, and X-Khronos) had not been definitively "broken" in the sense of being completely compromised, but they had been reverse-engineered to some extent by developers.

Various third-party libraries and tools claimed to generate these security tokens, allowing unofficial access to TikTok's API. This suggests that while the encryption mechanisms weren't completely secure against determined analysis, they still presented barriers to casual API access.

TikTok regularly updates these security mechanisms to counter reverse engineering efforts. They employ obfuscation techniques and frequently change implementation details, making it an ongoing cat-and-mouse game between TikTok's security team and those attempting to bypass their API protections.

Without more recent information, I can't comment on their current state, but it's likely that TikTok has continued to evolve these security measures while third parties have continued their efforts to bypass them.

Retry
L
many people claiming on github they can sell it via Telegram, is it scam?
Edit

Claims of selling TikTok API security tokens (X-Gorgon, X-Argus, X-Khronos) via Telegram should be treated with significant skepticism. There are several reasons to be cautious:

[LIST=1]
[*]Many of these offers are likely scams where you pay but receive nothing of value or something that doesn't work properly.
[*]Even if someone does provide a working solution, TikTok regularly updates their security mechanisms, so any "solution" would likely have a short shelf life before becoming obsolete.
[*]Some offerings might be malware disguised as token generators, putting your system and data at risk.
[*]Using unofficial methods to access TikTok's API violates their Terms of Service and could result in account bans or potentially legal issues.
[*]Legitimate reverse-engineering efforts are typically shared openly in developer communities rather than sold through private channels.
[/LIST]
While some developers have genuinely reverse-engineered aspects of TikTok's security implementation for research purposes, commercial offerings through Telegram channels promising "permanent" solutions should raise red flags

 

[shijin]

Have you tried using an API like TikTok-Api? It simplifies a lot of the complexity around headers and could save you some headaches.

 

[oatsactuary]

I was looking at yt-dlp which I know uses some internal APIs but noticed that

        aweme_detail = traverse_obj(
            self._call_api('multi/aweme/detail', aweme_id, data=urlencode_postdata({
                'aweme_ids': f'[{aweme_id}]',
                'request_source': '0',
            }), headers={'X-Argus': ''}), ('aweme_details', 0, {dict}))

https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/tiktok.py#L210-L215

... they send it empty, so not helping at all haha. But I also know you can use the search *without* been registered or logged in. That's what confused me.

 

[RaeAly]

I was looking at yt-dlp which I know uses some internal APIs but noticed that

        aweme_detail = traverse_obj(
            self._call_api('multi/aweme/detail', aweme_id, data=urlencode_postdata({
                'aweme_ids': f'[{aweme_id}]',
                'request_source': '0',
            }), headers={'X-Argus': ''}), ('aweme_details', 0, {dict}))

https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/tiktok.py#L210-L215

... they send it empty, so not helping at all haha. But I also know you can use the search *without* been registered or logged in. That's what confused me.

You don't actually need the signatures for all requests, very simple ones like aweme/detail probably don't need it. For search you might* since it's a harder endpoint (can have captcha I think).

Everything with TikTok API has varying levels of validation, for example, for some requests you can use a registered device that isn't activated, for others you need a properly activated device otherwise you get failed response or blank without signatures.

 

[oatsactuary]

My thoughts on the signatures though, can they really keep updating their signature keys if they need to support older versions of their app? I haven't researched extensively, but that's where my thoughts are. If an old version has the search functionality, and it's broken, it should be able to sign. But I could be completely wrong as well.

 

[RaeAly]

My thoughts on the signatures though, can they really keep updating their signature keys if they need to support older versions of their app? I haven't researched extensively, but that's where my thoughts are. If an old version has the search functionality, and it's broken, it should be able to sign. But I could be completely wrong as well.

The signatures change with security version changes (every 10-20 app updates), so if you use 1 app version you're fine never updating. The only issue is if the algorithm isn't actually accurate and they specifically patch that (Broken algorithms will work, for a bit), then you might get blank/invalid responses.

If you're worried about all that, I really would recommend just finding a mobile api based scraping service, the cost per request is usually extremely low.