Web hosting question

At the risk of sounding stupid I’d like to ask a question regarding web hosting.

Yesterday I created a thread in the Hire a freelancer section of BHW (https://www.blackhatworld.com/seo/scraping-job.1398869/). I describe how I want to scrape 8 websites. I was wondering… how much do people estimate it will cost me per month to host this much data?

And a second question…. how much do people estimate it costing Instagram per month to host all their data? (I know Facebook own Instagram but I’m just talking specifically about Instagram’s size).

In case you don’t want to click onto the other thread… here is the data I want to scrape….

IMDb.com entirely scraped (the whole website) and everything organised. Don’t worry about reviews or any articles. Other than the plots (synopsis) when it comes to the scraping as a general rule go with: don’t scrape anything longer than 10 words. So it’s more stats and data rather than things like reviews and articles.

RottenTomatoes.com. All the ratings for everything on the site.

Genius.com entirely scraped (the whole website) and everything organised. Don’t worry about the information given when you click on the lyrics or any articles. Other than the lyrics when it comes to the scraping as a general rule go with: don’t scrape anything longer than 10 words. So it’s more stats and data rather than things like reviews and articles.

TransferMarkt.com entirely scraped (the whole website) and everything organised. When it comes to the scraping as a general rule go with: don’t scrape anything longer than 10 words. So it’s more stats and data rather than things like reviews and articles.

TheFreeDictionary.com. All definitions for every word on the site. Note there are multiple definitions for each word from multiple sources that must be cited.

Dictionary.com. Phonetic spellings for every word on the site. E.g. wordplay becomes wurd-pley.

Thesaurus.com. Synonyms for every word on the site.

Wikipedia.com. Information on every person or thing in the database. Just to clarify - when I say I want information scraped from Wikipedia on every person or thing in the database - I mean *my* database that we build - what I don’t mean is every person or thing in Wikipedia’s database. I hope that’s clearer. E.g. Liverpool FC (football club), Leo Messi (footballer), Eminem (artist), The Blueprint by Jay Z (album), Reign Over Me (movie), Leonardo DiCaprio (actor) etc etc.

8 websites.

I hope someone can answer these queries. I appreciate BHW and it’s members and the invaluable things I’ve learnt here.

https://www.blackhatworld.com/seo/scraping-job.1398869/

As for your first question, I think that it will be much simpler and cheeper to get live data using public api from these sites, rather than scraping and keep all the data on your host.
This way, your site will be able to get data that is all the time updating, and not static data that miss new data.
An example for this APIs is Wikipedia:
https://www.programmableweb.com/api/wikipedia#:~:text=Because%20Wikipedia%20is%20built%20using,contained%20in%20the%20MediaWiki%20databases.
This way when your site need data, it gets it directly from Wikipedia database.

For that the simplest hosting plan can be enough, and you don't need an Instagram size of server.
Which leads me to your second question:
In 2013 Instagram had a huge jump in the amount of users and data, and moved to Facebook's servers.
Back then Instagram had around 150 million users. Last year they past 1B users.
Do your calculation...
If each user have 10 images of 1 mega each it means 10M Giga (and most users have much more than that).
They have farms of servers in few countries dedicated to the data of users.
Hope this comment help you a little.

monsieurlazar said:

As for your first question, I think that it will be much simpler and cheeper to get live data using public api from these sites, rather than scraping and keep all the data on your host.
This way, your site will be able to get data that is all the time updating, and not static data that miss new data.
An example for this APIs is Wikipedia:
https://www.programmableweb.com/api/wikipedia#:~:text=Because%20Wikipedia%20is%20built%20using,contained%20in%20the%20MediaWiki%20databases.
This way when your site need data, it gets it directly from Wikipedia database.

For that the simplest hosting plan can be enough, and you don't need an Instagram size of server.
Which leads me to your second question:
In 2013 Instagram had a huge jump in the amount of users and data, and moved to Facebook's servers.
Back then Instagram had around 150 million users. Last year they past 1B users.
Do your calculation...
If each user have 10 images of 1 mega each it means 10M Giga (and most users have much more than that).
They have farms of servers in few countries dedicated to the data of users.
Hope this comment help you a little.

Click to expand...

Thank you for responding to my thread. I have heard that getting live data through API intergration with these 8 websites would be the better way to go but I’m still just not sure. I’m not sure the websites (other than Wikipedia) will allow me to have all the data I need.

I assume Wikipedia is free API integration but what about the other 7 sites - do you think they will charge? And if so, how much am I looking at? I’ve done a little research but it looks like the data that’s provided isn’t the exact same as the data on the 8 websites I want to scrape.

I guess I just like the thought of keeping everything “in house”. Maybe I’m being foolish idk.

I'm not too familiar with all of the sites you asked to be scraped, but all of IMDB's data is available in tsv format so there's no much point in scraping it.
https://www.imdb.com/interfaces/

imdb: https://rapidapi.com/blog/how-to-use-imdb-api/
rotten-tomatoes: https://www.programmableweb.com/api/rotten-tomatoes-rest-api
genius: https://genius.com/discussions/161769-The-genius-api-is-now-public
This are example (in my previous comment was also Wikipedia.
Just Google the site name with public api and you will see.

This way you can just get the data from each site and present it on your site.
But it needs to be designed on your site, because you get only data, not the appearance.

Another way if your plan is only to get all the sites in one place is to embed the site in a window on your site.

roydan said:

I'm not too familiar with all of the sites you asked to be scraped, but all of IMDB's data is available in tsv format so there's no much point in scraping it.
https://www.imdb.com/interfaces/

Click to expand...

monsieurlazar said:

imdb: https://rapidapi.com/blog/how-to-use-imdb-api/
rotten-tomatoes: https://www.programmableweb.com/api/rotten-tomatoes-rest-api
genius: https://genius.com/discussions/161769-The-genius-api-is-now-public
This are example (in my previous comment was also Wikipedia.
Just Google the site name with public api and you will see.

This way you can just get the data from each site and present it on your site.
But it needs to be designed on your site, because you get only data, not the appearance.

Another way if your plan is only to get all the sites in one place is to embed the site in a window on your site.

Click to expand...

So as far as I can understand you are advising that we don’t scrape and build and host my own database? We just simply use API integration with those 8 websites and utilise their databases?

youngblacksmith said:

So as far as I can understand you are advising that we don’t scrape and build and host my own database? We just simply use API integration with those 8 websites and utilise their databases?

Click to expand...

Yes,Database for scrapping all these sites will coast much more than you can earn from it.
Good luck!

monsieurlazar said:

Yes,Database for scrapping all these sites will coast much more than you can earn from it.
Good luck!

Click to expand...

Ahh but you don’t know how I plan on monetising the data

Looks like I have much research to do.

Maybe I will go with public APIs, maybe I will go with scraping and building/hosting my own in house database, or maybe I will go with a mixture of both (I think probably this tbh). Thank you for your advice, much appreciated.