TikTok Private API Scraping - Cursor Limit??

[mr e]

I've built out a python script that scrapes hashtag API endpoint up to cursor 5000. (5000 videos).

When I go beyond this number, the json response contains the "has_more" field and it's set to 0.

Is this a server or client side cap? Anyone else have experience with this?

I'm not using any time stamped authentication tokens (X-gorgon, X-khronos, X-argus, etc.) because I haven't needed to so far. My current scraper can do 500 videos/min while scraping all the profile data I need. Although I have all the proper headers, parems, cookies. Haven't gotten rate limited, only stuck on this cap of 5000.

The goal is to build out my scraper to be able to scrape entire hashtag pages, millions of videos.

Any help would be appreciated, thank you.

 

[WebGeeksly]

What happens when you restart from video 499? It could be bot-circumvention solution by Tiktok, as most users will never watch 500 videos at the same time.. Consider doing a 10 sec pause before you continue wth video 500, maybe this works?

 

[DropVS]

what is your goal with it?

 

[mr e]

What happens when you restart from video 499? It could be bot-circumvention solution by Tiktok, as most users will never watch 500 videos at the same time.. Consider doing a 10 sec pause before you continue wth video 500, maybe this works?

I’ve tried with pausing or even just starting from cursor 5000 with a new session.

What happens is in the server response, the “has_more” field is set to 0. It could be a bot solution and require me to update my tokens/headers/parems but if I do that I lose the cursor session, as it’s tied to my current tokens/ip.

I was using a rooted emulator to build my script and intercept the requests, so maybe TikTok has different restrictions because it knows my device is rooted?

I just refuse to accept that it’s a server issue and you are limited to only 5000 videos per hashtag.

What do you think?

 

[mr e]

what is your goal with it?

Ideally scrape the entire hashtag page. So millions of videos per hashtag. Might be ambitious and I might hit other limits but there’s just no way this is a hard server cap. So I wanted to see if anyone had experience with this.

 

[Gogol]

Have you tried with some third party APIs first? May be try one of those in Rapid API, they have some free usage tiers as well so it will cost you nothing. If you can't do the same from there, chances are, you cannot do anything about the fact.

Heck, I would not even try to make a scraper like that myself, because of the maintainance hassle. I would just use a rapid api one instead.

 

[mr e]

Have you tried with some third party APIs first? May be try one of those in Rapid API, they have some free usage tiers as well so it will cost you nothing. If you can't do the same from there, chances are, you cannot do anything about the fact.

Heck, I would not even try to make a scraper like that myself, because of the maintainance hassle. I would just use a rapid api one instead.

Good idea, I’ll check that out.

I originally got started with my custom solution after using some ones on Apify, and they were limited to only 600 per hashtag. So I was able to almost 10x that.

I talked to some supposedly “expert” reverse engineer who told me he was able to get into the millions using IOS requests/calls.

Either way seems like I’m kind of at that point of serious work ahead or just use what I got…

 

[Gogol]

I talked to some supposedly “expert” reverse engineer who told me he was able to get into the millions using IOS requests/calls.

It may well be possible because mobile devices (specially ios) have higher "trust" than web. If you could somehow emulate an ios device request, it will probably give you more results. However, don't expect it to be easy. The requests won't be plaintext and I am sure there will be security mechanisms in place to stop you doing exactly what you are trying to do.

 

[mr e]

It may well be possible because mobile devices (specially ios) have higher "trust" than web. If you could somehow emulate an ios device request, it will probably give you more results. However, don't expect it to be easy. The requests won't be plaintext and I am sure there will be security mechanisms in place to stop you doing exactly what you are trying to do.

Yeah that makes sense. I was using a rooted android emulator for requests so I’m already using mobile API endpoints but like you said they’re likely lower trust and higher security.

 

[DropVS]

Ideally scrape the entire hashtag page. So millions of videos per hashtag. Might be ambitious and I might hit other limits but there’s just no way this is a hard server cap. So I wanted to see if anyone had experience with this.

But you Will make any month with that?

 

[redsdz94]

I've built out a python script that scrapes hashtag API endpoint up to cursor 5000. (5000 videos).

When I go beyond this number, the json response contains the "has_more" field and it's set to 0.

Is this a server or client side cap? Anyone else have experience with this?

I'm not using any time stamped authentication tokens (X-gorgon, X-khronos, X-argus, etc.) because I haven't needed to so far. My current scraper can do 500 videos/min while scraping all the profile data I need. Although I have all the proper headers, parems, cookies. Haven't gotten rate limited, only stuck on this cap of 5000.

The goal is to build out my scraper to be able to scrape entire hashtag pages, millions of videos.

Any help would be appreciated, thank you.

I'm having the same problem. Has there been any solution since?

 

[abbeychan]

There is no way to scrapy more, limiting up to 5,000

 

[ArnProxy]

This 5 000-item cutoff is imposed server-side by TikTok’s hashtag endpoint—it isn’t your script running out of steam. Beyond that cursor you’ll always see "has_more":0. A few workarounds folks use:

• Switch endpoints
  The “video_list” GraphQL endpoint (used by the web app’s infinite scroll) doesn’t enforce the same hard cap. If you reverse-engineer its query parameters (the cursor and count fields) you can page much deeper.
• Rotate your request context
  TikTok buckets hashtag results by “rank_range” (e.g. 0–499, 500–999, etc.). By tweaking that param in smaller windows you can stitch together more than 5 000 items.
• Use rolling time slices
  Instead of paginating from the very start, re-query with a date filter (e.g. video publish timestamp > T), moving T forward by 1–2 days each batch. You’ll avoid the 5 000 limit because each slice is “fresh.”
• Headless/browser automation
  Drive TikTok’s actual web UI (Playwright/Selenium). It will lazily load as you scroll, bypassing the endpoint cap at the cost of speed.

In short—it’s not on your side. Pick one of the above methods (most seamless is swapping to their GraphQL feed API) and you’ll reclaim those “missing” millions of posts.

 

[Anonymous865966]

Hello , I would like to ask a few question if we can discuss in private , can this script auto send messages to the scraped videos and does it also provide leads ? I would like us to work together on something

 

[GlobalProxy4g]

I've built out a python script that scrapes hashtag API endpoint up to cursor 5000. (5000 videos).

When I go beyond this number, the json response contains the "has_more" field and it's set to 0.

Is this a server or client side cap? Anyone else have experience with this?

I'm not using any time stamped authentication tokens (X-gorgon, X-khronos, X-argus, etc.) because I haven't needed to so far. My current scraper can do 500 videos/min while scraping all the profile data I need. Although I have all the proper headers, parems, cookies. Haven't gotten rate limited, only stuck on this cap of 5000.

The goal is to build out my scraper to be able to scrape entire hashtag pages, millions of videos.

Any help would be appreciated, thank you.

Yes, that’s a server-side cap, TikTok usually limits endpoints to 5000 results to prevent large-scale scraping. Even with proper headers and cookies, without dynamic authentication tokens (like X-gorgon, X-argus generated by the app), you’ll stay stuck at 5000. Many also use mobile proxies to avoid IP rate limiting and appear more legit, but bypassing the 5000 cap generally requires deeper reverse engineering of internal endpoints or grabbing dynamic tokens from a real device.