OccultKingdoms
BANNED
- Mar 11, 2013
- 5,542
- 9,844
Haha. I like your writing style. Lucky.sparks recommended your profile FYI. Your idea of automation looks similar to his.
26yo digital degenerate.
- Thread starter mindeswx
- Start date
DecadesMember
Banned - Multiple Rules Violations
- Jul 20, 2022
- 51
- 27
Hustling 247 u dont do any of that. Only online
Today is hard, tomorrow will be worse, but the day after tomorrow will be sunshine. - Jack Ma
optimalprime
Newbie
- Feb 21, 2022
- 33
- 8
Been toying with a very similar idea (albeit at a much smaller scale than you) and just wanted to suggest the r/tiktokthots subreddit as a source.Issue #1
TikTok, unlike instagram doesn't allow us to get the newest posts for a hashtag it only displays the content you 'should' like, so how the hell do we keep a steady flow of fresh content?
Let's do some
View attachment 218333
Let's say we would like to run 1k accounts, uploading 10 shorts per day, that's 10k shorts/day
1 short would consist of 3 TT videos, means we need 30k TT videos/day
A quick google search shows us that you should upload 2-3 videos if you want to become popular, but obviously not all users are going to do that, so let's just pretend that each user is uploading 1 video per day.
View attachment 218335
Means that we should monitor 30k TT accounts every day and download their videos, why not huh?
IMO with shorts the video quality plays a huge role, so having videos of kittens in a hot girls compilation would not be acceptable, the source channels should all be checked.
Getting the source channels
Step 1 - we manually create a source of 'sexy' tags to scrape like 'sexy' , 'bikini' , 'thightok' ,'thottok' (this was a good one) etc.
We are left with a decent list of accounts.
Step 2 - Scrape all the accounts from step 1 to gather their most used sounds and hashtags
Step 3 - We scrape all the sounds and hashtags for users - we have a large list of accounts
Step 4 - We scrape each accounts tiktok page getting some useful information like followers count, likes count, 10 last videos thumbnails and most importantly uploaded videos count
Step 5 - We discard the accounts that have uploaded less than 50 videos ( don't want to waste resources on monitoring half dead accounts) and
Step 6 - We download a pre-trained NSFW classifier https://github.com/notAI-tech/NudeNet and run the 10 images we have downloaded for all accounts through it,
Then we get the average tendency of images being 'sexy' and the number of NSFW videos per account ( from the 10 scraped earlier)
View attachment 218336
Step 7 - Now we could pretty accurately guess if the account is NSFW or not by NSFW count alone, let's say we would only take the accounts that have 4 NSFW videos, because we want quality and consistency, not someone who uploads one twerking video once in a while
Step 8 - Finally we have some source accounts we can use, although when going through some of the accounts I noticed some, let's say, not so appealing ones.
- Sometimes there's a guy in a video
- Sometimes the girl is ugly
- Sometimes the videos are inconsistant, for exmple, some videos are just a still image, and that's not what we need.
It is complicated to solve such problems programmatically because each of us understands beauty differently, so it's hard for the machine as well.
So I decided the last moderation step should be manual, the script takes one account ( that has 50+ videos and at least 4NSFW videos of last 10), opens the page and allows the user to do scroll the feed and decide manually if the account is worth to be used as a source.
Going through only several hundred of accounts made me realize that it is going to take way too long and I am just too lazy, so the script was sent to the VA, hopefully he has the same taste as I do in women
Next..
Set up a db, create a scraper that monitors all accounts every day, downloads new videos and creates the compilations.
Might turn up some new accounts to follow that didn't come up in the scrape.
What are you using to scrape btw? I experimented with a few tiktok scrapers on github without much success. Finally, I found https://rapidapi.com/neotank/api/tiktok-bulletproof/ and have had good results so far (though it will only scrape 30 of the user's tiktoks).
Looking forward to more of your journey
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
My previous post was deleted because I linked a page, so there might be a small gap for those who didn't read it in time.
@reaaski Funnily enough it was the @lucky.sparks thread that I stumbled upon when searching for gmail accounts that really triggered that urge to put my black hat on, to anyone who haven't yet his thread is a must read!
@optimalprime hope the following will answer your question.
Really getting carried away with this project and didn't have the time to update thread earlier, but I feel like I have to do it now, not to forget the important details. so here it goes.
Spoiler alert - there's going to be a lot of scraping.
Handpicking the user(source) accounts seems to be not sustainable, so we have to make a fully autonomous system, heck it's 2022 , with the technology we have access to it definitely is possible, it's only a question of persistence, and that I don't lack.
Will be writing everything in the same order that it happened, so you would also understand that you (at least I) can't get it right the first time always.
We've got a source of about 10k semi-filtered adult accounts, how can we find other related accounts?
Scrape all their posts to find the most frequently used hashtags and sounds, sounds good ay?
Akamai has joined the lobby
Tiktok has many annoying ways of blocking you without actually telling you that you're blocked, say you make a request to /@Account - everything loads, but it has no posts, okay let's continue?
But you open the very same account with your browser and there it is, a ton of posts
Solution- We work with requests until we get no posts returned from the account, once that happens we fire up our nifty browser, show Akamai that we're the real deal, solve captcha if needed, save the cookies and continue working with requests.
![copqlEb[1].png](https://www.blackhatworld.com/data/attachments/541/541212-2ba1ca8825bf5fb92374b60096090712.jpg?hash=K6HKiCW_X7 "copqlEb[1].png")
Notes, some countries work much better than others.
The captcha get's worse depending on how much you're spamming, starting from the slider one, to the 'select two identical images' and the rotation thing if you really fucked up.
Cool, we've got a fuckton of data scraped
![ben-stiller-eye-roll[1].gif](https://www.blackhatworld.com/data/attachments/541/541213-3db6d2d9f8c5a755c4091e716238ba4c.jpg?hash=PbbS2fjFp1 "ben-stiller-eye-roll[1].gif")
Let's analyze what interests us, music and the hashtags.
The top 50~ most frequently used sounds consist of 90% content we are interested in.
And the rest? - Not so much, turns out the sluts also just use whatever sounds are popular, well that's a bummer
Hashtags on the other hand are the real deal, we got at least 500 of hashtags that are primarily used only for adult-ish content.
Notes:
Is an 'accountant' some sort of a slang among younger generation?
About 50 tags are a variation of 'bikinni', I'll leave that one for your judgement.
Sound IDs are incremental, so we could theoretically always check the new sounds, BUT that also goes for any unpopular sound created by a random user.
Plans for the future -
When finally scraping users videos, get the sound IDs as well, so we can stay up to date with the newest trending sounds/tags in the niche.
Off to scraping those hashtags now!
Pretty much same as before, just with some improvements.
Thankfully when scraping posts from hashtags it shows you some data about the user, like post count, and user avatar, so we can skip the users with less than 40 videos uploaded AND we can fetch only the users avatar and guess it's gender with some le'pithon . So we don't need to scrape each users posts individually to decide whether he's/she's hot AND the gender. Saving some requests and traffic, who doesn't like that?
Then the other script takes only females, downloads her video thumbnails and guesses how NSFW the content is.
Notes:
Video thumbnail urls have an expiration token, video thumbnail - 7 days, but user avatar is much shorter. Like 1 day or something don't have it written down, but it's pretty short.
![fMuX4Rx[1].png](https://www.blackhatworld.com/data/attachments/541/541214-504b8b38078db492a6d28a34beb94f72.jpg?hash=UEuLOAeNtJ "fMuX4Rx[1].png")
A sudden self diagnosis of retardation.
So we are looking aaaalll over the place for NSFW tiktok accounts, and 90% of those accounts are doing OF
And it just so happens that you can actually have a link to your tiktok RIGHT IN YOUR DESCRIPTION
![5wBEpgx[1].png](https://www.blackhatworld.com/data/attachments/541/541215-b456142222f262f7a7cdf096c0cfd3a0.jpg?hash=tFYUIiLyYv "5wBEpgx[1].png")
Who could have thought that it could be so easy?
Definitely not fucking me.
What does that mean to us?
That's right - more scraping
Not going to get too much in to details of it, but I'm no specialist with OF, and AFAIK it does not allow you to search for users, so I used one of many available search engines.
Ba da beem ba da bow we've got ourselves a hefty list of OF models usernames.
Try to grab the tiktok urls from OF description - quite low percentage of girls have it there, only the advanced bitches with pimps from bhw do. (that's a business opportunity for you right here)
Try to match the usernames on OF to usernames on TT - no dice, very low accuracy and many false positives.
But know what most of them do have?
A link with a wall of all their other links, not sure how it's called, but here's the services ( and for the love of god BHW I'm not promoting these 'services' or selling anything)
allmylinks.com/username
linktr.ee/username
my.bio/username
So what we can do is - bruteforcing all of the above shortners, pretty simple stuff, just take {usernameOnOF} and make a request to all 3 shortners to see if there's a link to tiktok, and grab it.
![D0tvowH[1].gif](https://www.blackhatworld.com/data/attachments/541/541216-c9fdf58e0da75c3fbf8e64d046f6e446.jpg?hash=yf31jg2nXD "D0tvowH[1].gif")
That gets us a couple thousands accounts more.
That still does not satisfy our needs.
Those 50 page 'self improvement' audiobooks my brain has ingested while sleeping start kicking in..
...Think about why...
...Think about what...
So I slow down a bit and think why am I doing this.
To get the fresh NSFWish content from tiktok
But tiktok doesn't show the newest posts from hashtags like instagram does, or does it?
I do what I do best - a google search.
![yZCx1sa[1].png](https://www.blackhatworld.com/data/attachments/541/541218-d82de0a11f4de43a22b2b37b5fba40ab.jpg?hash=2C3goR9N5D "yZCx1sa[1].png")
One of those 'how to basic' videos pops up, showing the 'filter by date' working on the mobile app.
My exact reaction to that-
https://www.tiktok.com/@_holubrichard_/video/7106897609121336582
We're reverse engineering tiktok now. I guess
So buckle the fuck up
Artillery- memu,mitmproxy,frida
We're in
![drguKe6[1].png](https://www.blackhatworld.com/data/attachments/541/541219-d06156d66c2aa82a03808ece2d379eb7.jpg?hash=0GFW1mwqqC "drguKe6[1].png")
Spent way longer than I'd wish to admit, fiddling with the request, trying to replicate it but I had to admit that's out of my scope of knowledge.
Googling a bit about the x-gorgon headers only lead me to a few github repositories that both haven't been updated in 2 years~ , both of them claimed being able to generate the gorgon headers for the chinese tt alternative "douyin".
If anyone has any info on that - I'm all ears.
Notes:
If you think whether or not tiktok is tracking your X - the answer is yes. Yes they are.
A couple of hints: timezone, devices uptime, SIM cards MCC,MNC and phone number.
If we do everything properly we can scrape the data without being logged in, just what we wanted!
How it feels to tail -f in windows
Notes:
Sometimes on the first start the app likes to crash, just clean the applications data and it's working again!
Of course, stability is an issue with emulators, thank god the recent project I worked with is all about emulators, so it was rather easy to set it up properly and get it working on autopilot.
After some time the IP get's limited, after about 50~ hashtags scraped, just reset the machine, change the proxy IP and you're good to go.
P.s when using mitm as a proxy you can specify --mode upstream:IPORT - and the traffic will go out through the supplied proxy.
Cons of this method - I am only able to run 1 emulator at a time, because frida can only connect to one server AFAIK? If you know to specify emulator in frida, or have a tiktoks apk with ssl pinning patched lemme know, it would speed up the operation drastically!
So yeah that's like 4 days of work crammed up in to one post, tried to remember all the interesting bits and pieces, hope the memez helped reading it.
If someone want's to play around with those requests, or needs some of the data just ask. IT IS FREE . NOT FOR SALE chill bhw.
Sayonara
@reaaski Funnily enough it was the @lucky.sparks thread that I stumbled upon when searching for gmail accounts that really triggered that urge to put my black hat on, to anyone who haven't yet his thread is a must read!
@optimalprime hope the following will answer your question.
Really getting carried away with this project and didn't have the time to update thread earlier, but I feel like I have to do it now, not to forget the important details. so here it goes.
Spoiler alert - there's going to be a lot of scraping.
Handpicking the user(source) accounts seems to be not sustainable, so we have to make a fully autonomous system, heck it's 2022 , with the technology we have access to it definitely is possible, it's only a question of persistence, and that I don't lack.
Will be writing everything in the same order that it happened, so you would also understand that you (at least I) can't get it right the first time always.
We've got a source of about 10k semi-filtered adult accounts, how can we find other related accounts?
Scrape all their posts to find the most frequently used hashtags and sounds, sounds good ay?
Akamai has joined the lobby
Tiktok has many annoying ways of blocking you without actually telling you that you're blocked, say you make a request to /@Account - everything loads, but it has no posts, okay let's continue?
But you open the very same account with your browser and there it is, a ton of posts
Solution- We work with requests until we get no posts returned from the account, once that happens we fire up our nifty browser, show Akamai that we're the real deal, solve captcha if needed, save the cookies and continue working with requests.
Notes, some countries work much better than others.
The captcha get's worse depending on how much you're spamming, starting from the slider one, to the 'select two identical images' and the rotation thing if you really fucked up.
Cool, we've got a fuckton of data scraped
Let's analyze what interests us, music and the hashtags.
The top 50~ most frequently used sounds consist of 90% content we are interested in.
And the rest? - Not so much, turns out the sluts also just use whatever sounds are popular, well that's a bummer
Hashtags on the other hand are the real deal, we got at least 500 of hashtags that are primarily used only for adult-ish content.
Notes:
Is an 'accountant' some sort of a slang among younger generation?
About 50 tags are a variation of 'bikinni', I'll leave that one for your judgement.
Sound IDs are incremental, so we could theoretically always check the new sounds, BUT that also goes for any unpopular sound created by a random user.
Plans for the future -
When finally scraping users videos, get the sound IDs as well, so we can stay up to date with the newest trending sounds/tags in the niche.
Off to scraping those hashtags now!
Pretty much same as before, just with some improvements.
Thankfully when scraping posts from hashtags it shows you some data about the user, like post count, and user avatar, so we can skip the users with less than 40 videos uploaded AND we can fetch only the users avatar and guess it's gender with some le'pithon . So we don't need to scrape each users posts individually to decide whether he's/she's hot AND the gender. Saving some requests and traffic, who doesn't like that?
Then the other script takes only females, downloads her video thumbnails and guesses how NSFW the content is.
Notes:
Video thumbnail urls have an expiration token, video thumbnail - 7 days, but user avatar is much shorter. Like 1 day or something don't have it written down, but it's pretty short.
DING DONG
A sudden self diagnosis of retardation.So we are looking aaaalll over the place for NSFW tiktok accounts, and 90% of those accounts are doing OF
And it just so happens that you can actually have a link to your tiktok RIGHT IN YOUR DESCRIPTION
Who could have thought that it could be so easy?
Definitely not fucking me.
What does that mean to us?
That's right - more scraping
Not going to get too much in to details of it, but I'm no specialist with OF, and AFAIK it does not allow you to search for users, so I used one of many available search engines.
Ba da beem ba da bow we've got ourselves a hefty list of OF models usernames.
Try to grab the tiktok urls from OF description - quite low percentage of girls have it there, only the advanced bitches with pimps from bhw do. (that's a business opportunity for you right here)
Try to match the usernames on OF to usernames on TT - no dice, very low accuracy and many false positives.
But know what most of them do have?
A link with a wall of all their other links, not sure how it's called, but here's the services ( and for the love of god BHW I'm not promoting these 'services' or selling anything)
allmylinks.com/username
linktr.ee/username
my.bio/username
So what we can do is - bruteforcing all of the above shortners, pretty simple stuff, just take {usernameOnOF} and make a request to all 3 shortners to see if there's a link to tiktok, and grab it.
That gets us a couple thousands accounts more.
That still does not satisfy our needs.
Those 50 page 'self improvement' audiobooks my brain has ingested while sleeping start kicking in..
...Think about why...
...Think about what...
So I slow down a bit and think why am I doing this.
To get the fresh NSFWish content from tiktok
But tiktok doesn't show the newest posts from hashtags like instagram does, or does it?
I do what I do best - a google search.
One of those 'how to basic' videos pops up, showing the 'filter by date' working on the mobile app.
My exact reaction to that-
https://www.tiktok.com/@_holubrichard_/video/7106897609121336582
We're reverse engineering tiktok now. I guess
So buckle the fuck up
Artillery- memu,mitmproxy,frida
We're in
Spent way longer than I'd wish to admit, fiddling with the request, trying to replicate it but I had to admit that's out of my scope of knowledge.
Googling a bit about the x-gorgon headers only lead me to a few github repositories that both haven't been updated in 2 years~ , both of them claimed being able to generate the gorgon headers for the chinese tt alternative "douyin".
If anyone has any info on that - I'm all ears.
Notes:
If you think whether or not tiktok is tracking your X - the answer is yes. Yes they are.
A couple of hints: timezone, devices uptime, SIM cards MCC,MNC and phone number.
If we do everything properly we can scrape the data without being logged in, just what we wanted!
How it feels to tail -f in windows
Notes:
Sometimes on the first start the app likes to crash, just clean the applications data and it's working again!
Of course, stability is an issue with emulators, thank god the recent project I worked with is all about emulators, so it was rather easy to set it up properly and get it working on autopilot.
After some time the IP get's limited, after about 50~ hashtags scraped, just reset the machine, change the proxy IP and you're good to go.
P.s when using mitm as a proxy you can specify --mode upstream:IP
ORT - and the traffic will go out through the supplied proxy.Cons of this method - I am only able to run 1 emulator at a time, because frida can only connect to one server AFAIK? If you know to specify emulator in frida, or have a tiktoks apk with ssl pinning patched lemme know, it would speed up the operation drastically!
So yeah that's like 4 days of work crammed up in to one post, tried to remember all the interesting bits and pieces, hope the memez helped reading it.
If someone want's to play around with those requests, or needs some of the data just ask. IT IS FREE . NOT FOR SALE chill bhw.
Sayonara
Last edited:
optimalprime
Newbie
- Feb 21, 2022
- 33
- 8
That's amazing!!! Thanks for the update!My previous post was deleted because I linked a page, so there might be a small gap for those who didn't read it in time.
@reaaski Funnily enough it was the @lucky.sparks thread that I stumbled upon when searching for gmail accounts that really triggered that urge to put my black hat on, to anyone who haven't yet his thread is a must read!
@optimalprime hope the following will answer your question.
Really getting carried away with this project and didn't have the time to update thread earlier, but I feel like I have to do it now, not to forget the important details. so here it goes.
Spoiler alert - there's going to be a lot of scraping.
Handpicking the user(source) accounts seems to be not sustainable, so we have to make a fully autonomous system, heck it's 2022 , with the technology we have access to it definitely is possible, it's only a question of persistence, and that I don't lack.
Will be writing everything in the same order that it happened, so you would also understand that you (at least I) can't get it right the first time always.
We've got a source of about 10k semi-filtered adult accounts, how can we find other related accounts?
Scrape all their posts to find the most frequently used hashtags and sounds, sounds good ay?
Akamai has joined the lobby
Tiktok has many annoying ways of blocking you without actually telling you that you're blocked, say you make a request to /@Account - everything loads, but it has no posts, okay let's continue?
But you open the very same account with your browser and there it is, a ton of posts
Solution- We work with requests until we get no posts returned from the account, once that happens we fire up our nifty browser, show Akamai that we're the real deal, solve captcha if needed, save the cookies and continue working with requests.
View attachment 219035
Notes, some countries work much better than others.
The captcha get's worse depending on how much you're spamming, starting from the slider one, to the 'select two identical images' and the rotation thing if you really fucked up.
Cool, we've got a fuckton of data scraped
View attachment 219036
Let's analyze what interests us, music and the hashtags.
The top 50~ most frequently used sounds consist of 90% content we are interested in.
And the rest? - Not so much, turns out the sluts also just use whatever sounds are popular, well that's a bummer
Hashtags on the other hand are the real deal, we got at least 500 of hashtags that are primarily used only for adult-ish content.
Notes:
Is an 'accountant' some sort of a slang among younger generation?
About 50 tags are a variation of 'bikinni', I'll leave that one for your judgement.
Sound IDs are incremental, so we could theoretically always check the new sounds, BUT that also goes for any unpopular sound created by a random user.
Plans for the future -
When finally scraping users videos, get the sound IDs as well, so we can stay up to date with the newest trending sounds/tags in the niche.
Off to scraping those hashtags now!
Pretty much same as before, just with some improvements.
Thankfully when scraping posts from hashtags it shows you some data about the user, like post count, and user avatar, so we can skip the users with less than 40 videos uploaded AND we can fetch only the users avatar and guess it's gender with some le'pithon . So we don't need to scrape each users posts individually to decide whether he's/she's hot AND the gender. Saving some requests and traffic, who doesn't like that?
Then the other script takes only females, downloads her video thumbnails and guesses how NSFW the content is.
Notes:
Video thumbnail urls have an expiration token, video thumbnail - 7 days, but user avatar is much shorter. Like 1 day or something don't have it written down, but it's pretty short.
View attachment 219037
A sudden self diagnosis of retardation.
DING DONG
So we are looking aaaalll over the place for NSFW tiktok accounts, and 90% of those accounts are doing OF
And it just so happens that you can actually have a link to your tiktok RIGHT IN YOUR DESCRIPTION
View attachment 219038
Who could have thought that it could be so easy?
Definitely not fucking me.
What does that mean to us?
That's right - more scraping
Not going to get too much in to details of it, but I'm no specialist with OF, and AFAIK it does not allow you to search for users, so I used one of many available search engines.
Ba da beem ba da bow we've got ourselves a hefty list of OF models usernames.
Try to grab the tiktok urls from OF description - quite low percentage of girls have it there, only the advanced bitches with pimps from bhw do. (that's a business opportunity for you right here)
Try to match the usernames on OF to usernames on TT - no dice, very low accuracy and many false positives.
But know what most of them do have?
A link with a wall of all their other links, not sure how it's called, but here's the services ( and for the love of god BHW I'm not promoting these 'services' or selling anything)
allmylinks.com/username
linktr.ee/username
my.bio/username
So what we can do is - bruteforcing all of the above shortners, pretty simple stuff, just take {usernameOnOF} and make a request to all 3 shortners to see if there's a link to tiktok, and grab it.
View attachment 219039
That gets us a couple thousands accounts more.
That still does not satisfy our needs.
Those 50 page 'self improvement' audiobooks my brain has ingested while sleeping start kicking in..
...Think about why...
...Think about what...
So I slow down a bit and think why am I doing this.
To get the fresh NSFWish content from tiktok
But tiktok doesn't show the newest posts from hashtags like instagram does, or does it?
I do what I do best - a google search.
View attachment 219041
One of those 'how to basic' videos pops up, showing the 'filter by date' working on the mobile app.
My exact reaction to that-
https://www.tiktok.com/@_holubrichard_/video/7106897609121336582
We're reverse engineering tiktok now. I guess
So buckle the fuck up
Artillery- memu,mitmproxy,frida
We're in
View attachment 219042
Spent way longer than I'd wish to admit, fiddling with the request, trying to replicate it but I had to admit that's out of my scope of knowledge.
Googling a bit about the x-gorgon headers only lead me to a few github repositories that both haven't been updated in 2 years~ , both of them claimed being able to generate the gorgon headers for the chinese tt alternative "douyin".
If anyone has any info on that - I'm all ears.
Notes:
If you think whether or not tiktok is tracking your X - the answer is yes. Yes they are.
A couple of hints: timezone, devices uptime, SIM cards MCC,MNC and phone number.
If we do everything properly we can scrape the data without being logged in, just what we wanted!
How it feels to tail -f in windows
View attachment 219044
Notes:
Sometimes on the first start the app likes to crash, just clean the applications data and it's working again!
Of course, stability is an issue with emulators, thank god the recent project I worked with is all about emulators, so it was rather easy to set it up properly and get it working on autopilot.
After some time the IP get's limited, after about 50~ hashtags scraped, just reset the machine, change the proxy IP and you're good to go.
P.s when using mitm as a proxy you can specify --mode upstream:IP
Cons of this method - I am only able to run 1 emulator at a time, because frida can only connect to one server AFAIK? If you know to specify emulator in frida, or have a tiktoks apk with ssl pinning patched lemme know, it would speed up the operation drastically!
View attachment 219045
So yeah that's like 4 days of work crammed up in to one post, tried to remember all the interesting bits and pieces, hope the memez helped reading it.
If someone want's to play around with those requests, or needs some of the data just ask. IT IS FREE . NOT FOR SALE chill bhw.
Sayonara
The TikTok links on the OF page is genius and so obvious in retrospect lol. Hilarious also that the app itself just lets you filter by date.
So now that you got all the scraping done, what's the next step?
ffmpeg to join the vids together?
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
Yeah, sometimes you really wish you had brainstormed before doing a ton of work, but that's just how I work I guess, just sitting there and thinking ideas don't come. They come in the progress of doing something.
Yes, now that we've got a steady stream of fresh accounts just need to fine tune the user content scraper, probably going to take 20 video images instead of 10 to make the 'attractiveness' algo more accurate.
Also downloading videos is pretty tricky, to download the without the watermark you have to call the mobile endpoints, but the IPs get limited pretty quick, so we need to figure out how to warm up the mobile cookies.
When working in larger scale there's some things you have to plan ahead, like how strong the server has to be, how much storage will I need etc.
Thinking about just getting a 10TB HDD for local storage, and uploading older videos to unlimited drive since the videos wouldn't be accessed frequently, but it's a good idea to keep the older videos in case we want to throw them in to the mix later on.
And finally the youtube uploading itself, of course it would be great to simply connect it with google drive through zapier, so it would upload the new videos and we wouldn't have to use our proxy traffic for that, but still haven't read about the possible options to do that, anyone doing it with IFTT/zapier and has any insights
?
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
TT is really much harder to scrape compared to insta, but the amount of information returned is also staggering, take a look at the data we get back for 1 video
Tell me that's not every data scrapers wet dream. Reaally have to keep myself from digging deeper in to that beauty
Tell me that's not every data scrapers wet dream. Reaally have to keep myself from digging deeper in to that beauty
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
Finally, 2:30AM the scraping setup is done!
![B8NIH9D[1].png](https://www.blackhatworld.com/data/attachments/541/541518-ceade8f2038500482d59068938f756da.jpg?hash=zq3o8gOFAE "B8NIH9D[1].png")
So let's sum it up!
Template #1) We have a list of 400 tags to check, a VM is constantly running going from the last checked tag, searching for it and filtering by 'Yesterday' , we mitm the traffic and save all users usernames userIDs and profile img urls to the file.
Template #2 ) Takes the users scraped from step 1, download the profile image, make a prediction whether it's male or female, if we are sure it's a male then just add it to the database as male. If it is a female download 20 last video thumbnails and run the 'attractiveness' predictions and count the NSFW videos, add it to the db.
Template #3) Selects the last checked users from the database matching such criteria: videos > 40 , is female, sexy score > 20 or porn score > 20 or NSF videos count > 3 , we then get the user feed and grab the latest uploads tags, description, duration, music id and of course the video id.
Template #4) Selects the videos from db where downloaded is false and downloads it with yt-dlp, note that we can use the --cookies-from-browser argument to use the cookies collected by our browsers used in the previous steps ( remember that we fire up the browser to get the initial cookies for Akamai)
Notes:
So yeah, that's that, no memz this time, just wanted to lay this down and go get some rest
Btw, how long do you guys think the videos should be 10s, 15s, 60s?
So let's sum it up!
Template #1) We have a list of 400 tags to check, a VM is constantly running going from the last checked tag, searching for it and filtering by 'Yesterday' , we mitm the traffic and save all users usernames userIDs and profile img urls to the file.
Template #2 ) Takes the users scraped from step 1, download the profile image, make a prediction whether it's male or female, if we are sure it's a male then just add it to the database as male. If it is a female download 20 last video thumbnails and run the 'attractiveness' predictions and count the NSFW videos, add it to the db.
Template #3) Selects the last checked users from the database matching such criteria: videos > 40 , is female, sexy score > 20 or porn score > 20 or NSF videos count > 3 , we then get the user feed and grab the latest uploads tags, description, duration, music id and of course the video id.
Template #4) Selects the videos from db where downloaded is false and downloads it with yt-dlp, note that we can use the --cookies-from-browser argument to use the cookies collected by our browsers used in the previous steps ( remember that we fire up the browser to get the initial cookies for Akamai)
Notes:
- Damn some videos are so compact! Some 5 sec videos weight only 500kb , I guess the vertical orientation helps to save those pixels.
- We're using 3 types of proxies for this, sometimes you need a larger pool and sometimes the speed is more important, got to pick it accordingly!
- Because we get the last 30 videos each time we check the profile means we don't really have to check each profile every day, once a week or two should be enough, which means we can check more accounts/ save some traffic!
- yt-dlp is quite cpu intensive, so a separate machine will definitely be needed, probably it would be designated for downloading and combining the videos.
So yeah, that's that, no memz this time, just wanted to lay this down and go get some rest
Btw, how long do you guys think the videos should be 10s, 15s, 60s?
AV
Regular Member
- Feb 3, 2020
- 313
- 112
In my opinion 30-60s
D
Deleted member 1603965
Guest
Good luck mate.
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
Thought that the video downloading would slow us down, but since they are so compact we've downloaded 10k~ profiles per day which is plenty.
What I am pretty happy is the quality of videos, I'd say roughly 95% of them are exactly what we are looking for, the borderline NSFW girls. So we must have prepared the search filters properly.
It's still possible to add a check for each video in the future, to guess how 'sexy' it is, but not going to do it now, as it's already becoming a much larger thing than I wanted it to be.
ffmpeg is also kind of done. Was thinking of the best way to stitch several videos together and came up with this idea of making them pick sides by country, that way we could (hopefully) reach T1 countries.
Basically we got 2 videos, 2 watermarks and a sound, will have to get a list of some uncopyrighted sounds as well, anyone got any ideas?
Here's a couple of the videos so it would make more sense, bet the 90s kids will appreciate the choice of music hehe.
It's pretttty damn cringy if you'd ask me, but that's the content 'normal' people like.
https://www.youtube.com/shorts/ZLC58xN3no8https://www.youtube.com/shorts/u5pAUkucack(not selling anything here either btw)
Youtube accounts preparation
Starting with 100 accounts, logged in to all of them and started the warmup by collecting cookies from other googles products.
Notes:
It's strange that when downloading the videos you actually download not the 720x1280 video itself that you see on tiktok, but the original video that the user has uploaded, so dimensions ,fps and bitrate are all over the place.
Some accounts are reuploading the videos they have downloaded with the watermark, so we also download it watermarked.
What I am pretty happy is the quality of videos, I'd say roughly 95% of them are exactly what we are looking for, the borderline NSFW girls. So we must have prepared the search filters properly.
It's still possible to add a check for each video in the future, to guess how 'sexy' it is, but not going to do it now, as it's already becoming a much larger thing than I wanted it to be.
ffmpeg is also kind of done. Was thinking of the best way to stitch several videos together and came up with this idea of making them pick sides by country, that way we could (hopefully) reach T1 countries.
Basically we got 2 videos, 2 watermarks and a sound, will have to get a list of some uncopyrighted sounds as well, anyone got any ideas?
Here's a couple of the videos so it would make more sense, bet the 90s kids will appreciate the choice of music hehe.
It's pretttty damn cringy if you'd ask me, but that's the content 'normal' people like.
https://www.youtube.com/shorts/ZLC58xN3no8https://www.youtube.com/shorts/u5pAUkucack(not selling anything here either btw)
Youtube accounts preparation
Starting with 100 accounts, logged in to all of them and started the warmup by collecting cookies from other googles products.
Notes:
It's strange that when downloading the videos you actually download not the 720x1280 video itself that you see on tiktok, but the original video that the user has uploaded, so dimensions ,fps and bitrate are all over the place.
Some accounts are reuploading the videos they have downloaded with the watermark, so we also download it watermarked.
Last edited:
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
Eyes got sunburn so I had to take a few days off.
Yes, apparently that does happen and it does not feel good
Sat down today to get the uploading part going, but had to figure out what to do with the music..
Looked around youtubes own 'copyright free' library, but there's not a whole lot of songs and the quality isn't top notch, checked out a few other websites but the quality was more or less the same.
Remember that we need high quantities, so it has to be scalable, so I thought to take a look at the progress of AI generated sounds, that mf can do crazy shit these days, simply generated a couple thousands of 'tracks' and when attaching the audio to the video we can use the -ss parameter of ffmpeg to drop a couple seconds and milliseconds of the track, combine that with a couple of audio filters and you have nearly infinite number of unique tracks, not a whole lot different, but unique.
It's really interesting topic though, it's one thing for the AI to decide how 'attractive' a person is, by saying 'okay well the face has to be symmetrical, with X dimensions from eye to eye, eyes to the nose etc."
To judge the music on the other hand - it's much more complicated, there are so many different genres different types of people enjoy, heck even the same person can enjoy one song in the morning but hate it in the evening.
There are some truly terrifying fart sounds in some of the songs, but some of them are actually really nice.
Here's a few examples:
https://youtube.com/shorts/Mu_uhNSgZi0?feature=sharehttps://youtube.com/shorts/S7koxmxjSRI?feature=sharehttps://youtube.com/shorts/8E8ylu80TqE?feature=share
I think that the sounds are the weakest link now, it just doesn't 'click' , but what do I know, maybe gen Z will eat it up as "rad shiii"
Another idea is to use the original sound just put a filter on it, or just putting these 'tracks' on top of the originals, but googles sounds identification system is definitely hard to trick and copyright strikes is the last thing we want, so we'll have to see...
Yes, apparently that does happen and it does not feel good
Sat down today to get the uploading part going, but had to figure out what to do with the music..
Looked around youtubes own 'copyright free' library, but there's not a whole lot of songs and the quality isn't top notch, checked out a few other websites but the quality was more or less the same.
Remember that we need high quantities, so it has to be scalable, so I thought to take a look at the progress of AI generated sounds, that mf can do crazy shit these days, simply generated a couple thousands of 'tracks' and when attaching the audio to the video we can use the -ss parameter of ffmpeg to drop a couple seconds and milliseconds of the track, combine that with a couple of audio filters and you have nearly infinite number of unique tracks, not a whole lot different, but unique.
It's really interesting topic though, it's one thing for the AI to decide how 'attractive' a person is, by saying 'okay well the face has to be symmetrical, with X dimensions from eye to eye, eyes to the nose etc."
To judge the music on the other hand - it's much more complicated, there are so many different genres different types of people enjoy, heck even the same person can enjoy one song in the morning but hate it in the evening.
There are some truly terrifying fart sounds in some of the songs, but some of them are actually really nice.
Here's a few examples:
https://youtube.com/shorts/Mu_uhNSgZi0?feature=sharehttps://youtube.com/shorts/S7koxmxjSRI?feature=sharehttps://youtube.com/shorts/8E8ylu80TqE?feature=share
I think that the sounds are the weakest link now, it just doesn't 'click' , but what do I know, maybe gen Z will eat it up as "rad shiii"
Another idea is to use the original sound just put a filter on it, or just putting these 'tracks' on top of the originals, but googles sounds identification system is definitely hard to trick and copyright strikes is the last thing we want, so we'll have to see...
ColdAndDark
Junior Member
- Nov 27, 2018
- 136
- 40
Nothing I love more than the good old hussle.
Seems like your life is one rollercoaster ride.
Seems like your life is one rollercoaster ride.
D
Deleted member 731332
Guest
Lol Gen Z is going to love that lmao
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
Ooookay, so today I had some time to sit down and check how the things are going, and they aren't going that great!
As usual with shorts, just like on tiktok videos go 'viral' randomly, you could say that the account is shitty (which definitely is also true) and it is 'shadowbanned' , but for some magical reason views do explode sometimes, as you can see in this example the video that got views is just like all other, it doesn't have a title/description either.
There was a bug in the code which caused 1 video getting uploaded to all of the channels , so you could say they're all fucked now ¯\_(ツ)_/¯
Was looking for a solution that would help tracking all the account stats, but could not find anything, so a simple dashboard was born, do we really need to know this information? -
Probably not, but it surely is nice ho track.
I really thought this would work out faster, and sadly I have another project to work on that is more important, BUT now the next steps are to leave more accounts on the warmpup/cookie farming stage, and I will leave the active ones running to monitor for bugs to fix and make sure it's ready for scaling!
Till next time you sexy bastards!
As usual with shorts, just like on tiktok videos go 'viral' randomly, you could say that the account is shitty (which definitely is also true) and it is 'shadowbanned' , but for some magical reason views do explode sometimes, as you can see in this example the video that got views is just like all other, it doesn't have a title/description either.
There was a bug in the code which caused 1 video getting uploaded to all of the channels , so you could say they're all fucked now ¯\_(ツ)_/¯
Was looking for a solution that would help tracking all the account stats, but could not find anything, so a simple dashboard was born, do we really need to know this information? -
Probably not, but it surely is nice ho track.
I really thought this would work out faster, and sadly I have another project to work on that is more important, BUT now the next steps are to leave more accounts on the warmpup/cookie farming stage, and I will leave the active ones running to monitor for bugs to fix and make sure it's ready for scaling!
Till next time you sexy bastards!
D
Deleted member 731332
Guest
I hate those small things in code that get overlooked which turn out to be a big deal. Hope this continues!
jaycewarrior
Senior Member
- Jun 28, 2017
- 917
- 503
This is a quiet complicated setup haha. I created a simple browser macro that scroll thro a specific hashtag and gather all the links of the vids save them in a file and download them.
If you would like to find active onlyfans accounts with link in their description use this dork: site:Onlyfans.com info:Last seen Today contains:linktree
Publishing babes vids on shorts is a nightmare, i had one video goes viral and every next upload is just few views to none on multiple accounts.
Also, I see you mixing two vids in one without adding changes to them which as far of my knowledge abt youtube algo can easily read it as an compilation video and distinguish each of the two clips used to identify them separately which could lead to a detection of spam pattern if those clips are been used repetitively.
Just keep in mind Shorts algo works in an unpredictable way.
Good luck!
If you would like to find active onlyfans accounts with link in their description use this dork: site:Onlyfans.com info:Last seen Today contains:linktree
Publishing babes vids on shorts is a nightmare, i had one video goes viral and every next upload is just few views to none on multiple accounts.
Also, I see you mixing two vids in one without adding changes to them which as far of my knowledge abt youtube algo can easily read it as an compilation video and distinguish each of the two clips used to identify them separately which could lead to a detection of spam pattern if those clips are been used repetitively.
Just keep in mind Shorts algo works in an unpredictable way.
Good luck!
mindeswx
Regular Member
- Dec 5, 2016
- 362
- 594
I also went that route of scraping hashtags at first, but the thing is - the newest videos are not at the top as it is with instagram, so to get a reasonable number of posts we have to filter by the newest ones.
Thanks for the dork idea, haven't thought about that!
Yes, the videos are getting combined, but that's why we have to scrape so many videos from tiktok, because no video is getting put in to the complication twice, so basically 2 tiktok videos = 1 YT short, but yeah it's definitely strange how the algo works
jaycewarrior
Senior Member
- Jun 28, 2017
- 917
- 503
Oh i see! you need newest videos. Scraping vids under hashtags works for me since i need viral vids that proven to be effective so i get a higher chance the shorts algo pick-it up.I also went that route of scraping hashtags at first, but the thing is - the newest videos are not at the top as it is with instagram, so to get a reasonable number of posts we have to filter by the newest ones.
Thanks for the dork idea, haven't thought about that!
Yes, the videos are getting combined, but that's why we have to scrape so many videos from tiktok, because no video is getting put in to the complication twice, so basically 2 tiktok videos = 1 YT short, but yeah it's definitely strange how the algo works
Try this: in ffmpeg add color filter and mirroring, add multiple layers of audio, add a blurred picture at middle or the end of the clip (0.7s), add a small text layer. This should make the video look unique enough in the eyes of the algo.
D
Deleted member 731332
Guest
How has this fared after the recent update with yt shorts?