[TIKTOK SCRAPER Profile & Tags] (Testing Journey) NO API

Hey everyone, (I'm not selling anything, I'm not advertising anything, this is my own personal research that I wanted to share with you all)

One more time I'm here with one of my crazy ideas, so recently I decided to scrape some Tik-tok videos for a new youtube journey that I want to start (btw I'm keeping away from music videos since they get easily copy strikes). So I went into my dear friend google, and I started researching:

Day 1 findings: (FAILED)
I used selenium to create a bot scraper for Tiktok, I totally failed, Tiktok has some advance technology protecting their front end as well as back-end network packages.

Day 2 findings: (FAILED)
I decided to try a different way of scraping it, unfortunately it didnt work, I also found out that Tiktok uses four plus servers to host their videos, the mp4 links are protected by their walls of <divs> and other crappery. their robot.txt somehow blocks every single scraper that I have used.

Day 3 findings: (WIN)
I found a git that allows the download of individual videos (1 by 1), the good thing is that the videos download without watermark, the bad thing is that you can only do 1 by one, no bulk download no bulk watermark helper, I found another git, but unfortunately they use a online web server that deals with videos through their own personal API (which I believe they change everyday due to getting banned for too many video downloads)

Day 4 findings: (Failed)
I decided to not use any of those, since I dont want to depend on a website that could go down tomorrow, due to their api, nor use somebody elses code, so I decided that I would start coding my own scraper no matter what.

Day 5 findings: (Winner Winner Chicken Dinner):
I found a way of scraping TikTok, I can scrape up to 256 videos in around 4 minutes without touching my keyboard or mouse. I have also developed a simple python script that removes the watermark from the video.

Day 6 findings: (nice ride) (Scraping by PROFILES and TAGS)
I can now scrape in bulk, download in bulk, and I'm improving my own personal script to remove the watermarks in bulk...

Next Week:
I'm planning to attach both of my scripts, with my adobe after effects plugins in order to automate the creation of videos with a limit video time of 5 minutes, meaning that I could create up to 80 videos of 5minutes each in 10 minutes ready to be uploaded to youtube.

I hope TIKTOK doesnt get mad with me, but this bastards really made it hard for us to scrape their site.

SPOILER

Few Things to Highlight -

1.-I didnt want to use already made code out there first of all because it doesnt work properly, and I'm sure tiktok already is studying those public git out there.
2.-I didnt want to use any online service.
3.-I didnt want to use any sort of api for this project, therefore this process cannot get patch unless they change the looks of the tiktok site.

Feel free to ask questions.

Question: why would I care what you are doing with your free time?

benxjugger said:

Question: why would I care what you are doing with your free time?

Click to expand...

Nice Question, it seems as you already do, you already landed on my precious post... and even replied...

lol in all honesty, I'm trying to share something nice with the gang, but again you dont need to read it or look at it if you dont want, best of luck

selenium .....Your first mistake was this as there are many better options that i dont believe you looked up prior to finding a premade git.

benxjugger said:

Question: why would I care what you are doing with your free time?

Click to expand...

Wow I bet you’re fun at parties!
Welcome to BHW where members share their journeys in different fields.

UnknownJones said:

selenium .....Your first mistake was this as there are many better options that i dont believe you looked up prior to finding a premade git.

Click to expand...

Hi Jones, I'm sorry bro, could you please tell me what the better options are, I'm quite new to scraping, and I thought that selenium was the actual best, I even configured everything properly such headers, locations, browser type, device type, etc. But still I didn't have any success.

Paukerin said:

Wow I bet you’re fun at parties!
Welcome to BHW where members share their journeys in different fields.

Click to expand...

Just don't waste your energy with him bro, don't mind him.

Paukerin said:

Wow I bet you’re fun at parties!
Welcome to BHW where members share their journeys in different fields.

Click to expand...

And I bet you know horse shit about Tiktok and their encryptions so why worry.

OP, So what is this journey about and how's it helpful?
You're taking someone else code from Github then put it all together to make a scraper? Outstanding. Please let me know when you will reverse xlog yourself.

benxjugger said:

And I bet you know horse shit about Tiktok and their encryptions so why worry.

OP, So what is this journey about and how's it helpful?
You're taking someone else code from Github then put it all together to make a scraper? Outstanding. Please let me know when you will reverse xlog yourself.

Click to expand...

Lol this guy...
I think you didn't read the post, I mentioned that I didn't use anyone's gits... Its coded form scratch. Let's just end it there, thanks for coming, good luck.

justmeseo said:

Hi Jones, I'm sorry bro, could you please tell me what the better options are, I'm quite new to scraping, and I thought that selenium was the actual best, I even configured everything properly such headers, locations, browser type, device type, etc. But still I didn't have any success.

Just don't waste your energy with him bro, don't mind him.

Click to expand...

Look into request based methods. There are alot of open soruce mods for python and scripts that are/were public which you need to tweak. Obviously wont happen overnight but if you can reverse engineer this, can do far more sites in the future.

Appreciated I will look into it and report back.

UnknownJones said:

Look into request based methods. There are alot of open soruce mods for python and scripts that are/were public which you need to tweak. Obviously wont happen overnight but if you can reverse engineer this, can do far more sites in the future.

Click to expand...

So you share the way you failed but skip the way you win dude?

Here a clue for others understands PHP.


All it does, download tiktok video with no watermark in PHP language.

Tiktok there self hold watermark and none watermark video versions.


Only a example from online git project .

Not sure what you mean, I'm holding about 3000 videos in different niche, I'm building an automation to parse everything through after effects, and thereafter render to YouTube, so in terms of progress I can't update because I ain't got much. My main goal is to start seeing results with youtube, and kind of promote other services through those accounts.

lionK said:

So you share the way you failed but skip the way you win dude?

Click to expand...

Readarrow I appreciate your comment but as explain I have seen this code not only on php but also in python, as I said I'm trying to build something without using any type of api (please read my post), as you may know apis could get easily limited or cancelled if they are being exploited.

I don't want to be depending of a system that limits my scraping functionality.

Also I can currently scrape in bulk by tag and profile ID, and I remove watermark in bulk through my own code.

redarrow said:

Here a clue for others understands PHP.

Code:

<?php
require_once __DIR__ . "/constants.php";
require_once INC_DIR . "functions.php";
if (isset($_GET['url']) && !empty($_GET['url'])) {
    $url   = get_param('url');
    $key   = get_param('key');
    $api    = new \TikTok\Api(["license-key" => get_option('license_key')]);
    $download = new \Sovit\TikTok\Download();
    $videos = $api->getVideoByUrl($url);
    if ($videos && isset($videos->items) && !empty($videos->items)) {
        $video = $videos->items[0];
        switch ($key) {
            case "video":
            default:
                if (isset($video->video->downloadAddr) && !empty($video->video->downloadAddr)) {
                    $src      = $video->video->downloadAddr;
                    $download->url($video->video->downloadAddr, $video->id, 'mp4');
                }
                break;
            case "no-watermark":
                $noWatermark = $api->getNoWatermark($url, $video->id);
                $download->url($noWatermark->url, $video->id.'-no-watermark', 'mp4');
           


                break;
            case "music":
                if (isset($video->music->playUrl) && !empty($video->music->playUrl)) {
                    $src      = $video->music->playUrl;
                    $download->url($video->music->playUrl, $video->music->id, 'mp3');
                    force_download($src, $filename);
                }

                break;
        }
    }
}
exit;

All it does, download tiktok video with no watermark in PHP language.

Tiktok there self hold watermark and none watermark video versions.

Code:

            case "no-watermark":
                $noWatermark = $api->getNoWatermark($url, $video->id);
                $download->url($noWatermark->url, $video->id.'-no-watermark

Only a example from online git project .

Click to expand...

About you being new in scraping, I've been doing this for a while now. Let me share some tips and tricks that'll work in general.

1. Never use a public tool like Selenium. Unless you're 100% sure it's a small project or the website doesn't care, and you have infinite resources.
2. Never use emulation. I have learned it from a senior developer who was scraping from the late 90s. He had a motto when it came to general scraping, "use basic requests." It'll help you with resource usage, speed, etc.
3. When a website acts rough on you by forcing you to fill in weird javascript functions (eg. PornHub), try using NodeJS and do the function using direct evaluation. Don't waste your time recreating the wheel in another language.
4. Try using text search, then Regex, then libraries. Fallback assorted. It'll help you with your project speed.
5. Try using fewer libraries, and stay as close to native as possible.
6. If you have time, I suggest investing your time in reverse-engineering the APIs as it's faster compared to HTML parsing. For sites running on JavaScript, APIs will be a better investment overall.

TL;DR, don't slow your process by wasting time on emulation nor finding out how a library works and slow things down in the process.

Wow, that was very helpful,thank you so much,and by the way you are 100% right in your points.

I have done from point 1 to point 4, even regex was useless towards the loading rendering and lazy loading of tiktok.

Tiktok is currently using react js, to stop the loading of videos in the site and therefore as you hover through the videos, it covers the video link of the one you hover before.

If that's the case, I'm reverse engineering how they are hiding this links with react native, all I have to do is to activate this videos automatically for the entire page (this means a javascriot code to scroll all the way down and a javascriot to scroll all the way up, and active hovering for each of this videos) the funny thing is that html won't work for this since their react js framework, this will have to be done using emulation at the end of the day yes or yes, since the process could only work on the front end.

But thank you so much for your thoughts, if selenium might not be the best for this sort of deal, what system would you recommend?

The Undefined said:

About you being new in scraping, I've been doing this for a while now. Let me share some tips and tricks that'll work in general.

1. Never use a public tool like Selenium. Unless you're 100% sure it's a small project or the website doesn't care, and you have infinite resources.
2. Never use emulation. I have learned it from a senior developer who was scraping from the late 90s. He had a motto when it came to general scraping, "use basic requests." It'll help you with resource usage, speed, etc.
3. When a website acts rough on you by forcing you to fill in weird javascript functions (eg. PornHub), try using NodeJS and do the function using direct evaluation. Don't waste your time recreating the wheel in another language.
4. Try using text search, then Regex, then libraries. Fallback assorted. It'll help you with your project speed.
5. Try using fewer libraries, and stay as close to native as possible.
6. If you have time, I suggest investing your time in reverse-engineering the APIs as it's faster compared to HTML parsing. For sites running on JavaScript, APIs will be a better investment overall.

TL;DR, don't slow your process by wasting time on emulation nor finding out how a library works and slow things down in the process.

Click to expand...

And I get it, there is not need to recreate the wheel, my goal is not to recreate it, but make it better. Hope we could partner together, perhaps have a chat, I would love to hear your ideas.

I will keep you posted with any updates otherwise.

justmeseo said:

Wow, that was very helpful,thank you so much,and by the way you are 100% right in your points.

I have done from point 1 to point 4, even regex was useless towards the loading rendering and lazy loading of tiktok.

Tiktok is currently using react js, to stop the loading of videos in the site and therefore as you hover through the videos, it covers the video link of the one you hover before.

If that's the case, I'm reverse engineering how they are hiding this links with react native, all I have to do is to activate this videos automatically for the entire page (this means a javascriot code to scroll all the way down and a javascriot to scroll all the way up, and active hovering for each of this videos) the funny thing is that html won't work for this since their react js framework, this will have to be done using emulation at the end of the day yes or yes, since the process could only work on the front end.

But thank you so much for your thoughts, if selenium might not be the best for this sort of deal, what system would you recommend?

Click to expand...

justmeseo said:

And I get it, there is not need to recreate the wheel, my goal is not to recreate it, but make it better. Hope we could partner together, perhaps have a chat, I would love to hear your ideas.

I will keep you posted with any updates otherwise.

Click to expand...

Awesome, drop me a message here so we can discuss this further. There is always a way to overcome the emulation, as it's super resource intensive.

i have one question , what are you trying to do i mean with videos that you are trying to download in bulk?

Sent

The Undefined said:

Awesome, drop me a message here so we can discuss this further. There is always a way to overcome the emulation, as it's super resource intensive.

Click to expand...

I specialise on social media marketing, I have few channels that lack some content, thankfully there are people out there who post valuable content for my niche.
I could say with assurance that my niche has been untachable for many years, and it's not very well known by many, so my videos will really hit my target audience.

TLDR:
MONETIZATION on youtube videos

MgicalaEarner said:

i have one question , what are you trying to do i mean with videos that you are trying to download in bulk?

Click to expand...