ooh lots of interesting stuff to respond to here--
the ixxx/tubegalore scripts actually show really good thumbs, it's why they've survived so long IMO. It doesn't matter what other features you have, your site won't be competitive with theirs until you build up some data and can compete with their thumbs
I was also scraping true views/ratings on my sites in the early days. It helped me to get ahead on the thumbs game when I had no other data to rely on. In the end, keeping the scraper up to date was too much of a pain and I had plenty of my own data for good thumbs, so I scrapped the feature. It didn't bother my users (or at least, my metrics), they don't seem to care about anything but sexy thumbs. These days it might be easier to maintain a scraper because the affiliate program APIs and resources are getting a lot better in general. I was mostly having to scrape actual video pages. Totally sucked.
I also had thumb slideshows when I launched, for the same reason as you--an advantage over the incumbents. Users like it.
That's a large DB. You don't really need that many videos. Not gonna lie all my sites largely inflate the video counts, it makes my life easier and doesn't affect my users really. Such a large DB could make it harder to scale your site (remember, users will search "teen" and it will match 1/4 your database...that could be 10m results returned from the query. It's a big query.) In general I hope you've thought a lot about how your site would behave if it were getting 10+ hits/second. I did pretty (really) bad with scalability on my first script attempt and so had to make incremental improvements along the way to get the site handling heavy traffic.
10k is a workable budget but you'll have to spend it right, I don't think its enough to get away with very many mistakes. I won't say anything about how to spend it lol, part of the game for me was buying from a TON of different sellers/brokers, wasting money and domains, until I had a set of very strong trusted sellers to buy from...and then further having to test buying strategies like anchor texts, combinations of link types, sellers, etcBut google won't fuck you or de-index you because of embeds, so at least you don't have to worry about that. Google will only fuck you for spam--which isn't to say you shouldn't give spam a shot now and then!
Thanks for always replying to my post in every thread! I really appreciate it!
Could you elaborate on this point? All the tubes give csv dumps/rss feeds with thumbnails. Including the preferred thumb that loads in the search results before the user hovers over the card to activate the slideshow. Is there a problem with the standard stuff they give you?
Can you elaborate on this? I already have the scraping for all tubes coded & it works. One of the last parts I have to code is the backend dashboard to monitor the scraper. I plan on giving each tube a dozen "quality control" links. They will get checked daily. If the scraper shows results that don't match the saved defaults it will show on the dash. Example: View count integer increasing is healthy. When the scraper can't find the field on the majority on control links it shows on the dash.
Blows my mind that the big company that controls all these sites doesn't use this obvious feature.
My calculation put the SQL DB size on just under 100gb. A lot of fields won't be needed for search so I won't feed them into elasticsearch. elasticsearch can easily search a 70-80gb database with blazing speed. I worry more about if duplicates ruin the user experience, bc the same promo vids are uploaded to all the big tubes by the studios.
Just an example. I just turned 30 and am from a first world country with a pretty good job. If I see results I have the cash to feed the growth. My worry is more that google doesn't like that my site is 100% embeds and it gives me no traffic regardless of what I build in links.
Maybe I am missing something. I know established domains and hosting costs money as well as building the frontend/backend of the site but are you spending money on anything else? Just a little confused about where the 30K+ is going.
Thanks for always replying to my post in every thread! I really appreciate it!
Could you elaborate on this point? All the tubes give csv dumps/rss feeds with thumbnails. Including the preferred thumb that loads in the search results before the user hovers over the card to activate the slideshow. Is there a problem with the standard stuff they give you?
Can you elaborate on this? I already have the scraping for all tubes coded & it works. One of the last parts I have to code is the backend dashboard to monitor the scraper. I plan on giving each tube a dozen "quality control" links. They will get checked daily. If the scraper shows results that don't match the saved defaults it will show on the dash. Example: View count integer increasing is healthy. When the scraper can't find the field on the majority on control links it shows on the dash.
Blows my mind that the big company that controls all these sites doesn't use this obvious feature.
My calculation put the SQL DB size on just under 100gb. A lot of fields won't be needed for search so I won't feed them into elasticsearch. elasticsearch can easily search a 70-80gb database with blazing speed. I worry more about if duplicates ruin the user experience, bc the same promo vids are uploaded to all the big tubes by the studios.
Just an example. I just turned 30 and am from a first world country with a pretty good job. If I see results I have the cash to feed the growth. My worry is more that google doesn't like that my site is 100% embeds and it gives me no traffic regardless of what I build in links.
By "good thumbs" i don't mean the best thumb in a particular video's flipbook. I mean good thumbs given a query. Someone searches "anal" at least some of the thumbs should have a hot chick with a dick in her ass. Someone searches "blowjob" as many thumbs as possible should show a hot chick looking sexy with a dick in her mouth. A thumb that gets clicked on a lot when it shows up in "blowjob" might suck when it shows up in "anal"
solving that problem is pretty much the magic of running these sites. If you get it right you can make money, if you don't you'll be outcompeted.
A health check for your scraper is a good idea and yeah writing the scraper is easy but remember if your site is successful you gotta keep it running for YEARS. You are going to get incredibly fucking tired of checking that health check every day and you will be surprised how often something will be broken--and fixing a broken scraper is pure grunt work that you will come to hate with passion. It's just an annoying component to maintain long term
As far as elasticsearch...a large DB is definitely feasible but if you can massively cut CPU usage by dropping the size of the DB and keep your users just as happy...worth considering?
And oh by the way if you cut that DB size by removing some sites from your index, well you just did a lot to solve the duplicates problem, now didn't you? In fact why bother having every single site on your site if 90% the content is duplicate anyway?
Also by the way if your elasticsearch query uses views/ratings then these fields will have to be indexed which means your scraper will have to continually update index fields in ES, and ES can't update indexed fields without reindexing the document, which causes ES to bust its query cache and also forces segment merges in the underlying lucene processes (which is BAD), so right now you've got a situation where you either can't use views/ratings in your ranking function or you have to reindex documents all day every day meaning queries will never be cached and you will be merging segments all day every day so you might be happy to have a small DB, it will help this problem![]()
Throwing money at a site and losing it all is just part of the game but again, don't worry about the embeds, the only thing that will get you penalized is black-hat tactics
I just turned 31![]()
You cant hide it no matter what you do.