Instagram Database with over 100million records. How to store all that data?

nighthawk12 said:

Hey man, thanks for the suggestion i will start looking into to it!
I was thinking of using an oracle cloud database, they told me is fast and I can filter it as i want.

Click to expand...

nighthawk12 said:

Yes, most of my emails are verified. But what I do is when I use them for my needs, I always verify the batch.
To make audiences, cold emails. bulk emails etc.

It depends because I don't use all of them actually, only the ones that meet my requirements.

Click to expand...

nighthawk12 said:

You are correct, I use them for many things building audiences for ads, cold emails, bulk emails. Really depends. The reason why i want to save them all is that its faster since that Instagram data is Gold if you use it correctly, don't want to confuse all the files.
My process is getting the raw file, saving it in the main database, running a query to remove duplicates, save only unique values. Then extract the unique categories(niches) I need.

But i was speaking with an expert, probably going to make an Oracle database. Fast, simple and can access from everywhere since it's in the cloud. Better than local xD

Click to expand...

nighthawk12 said:

Hey guys if you are interested more in my IG scraping journey, please check this other Thread. https://www.blackhatworld.com/seo/instagram-data-extraction-scraping.1467814/#post-16015197

Click to expand...

proxygo said:

@nighthawk12 little tip - you can reply to 4 people in 1 post like this.

Click to expand...

Tasker18 said:

Mongodb or any file system based DB will help
convert and import the data into it and later access via DB tools

Click to expand...

Johnpeterburs said:

Start doing sentimental analysis for Instagram or Twitter using apache tools particularly from Nifi. It does synchronize, channelize and save data to further ingestion.

Click to expand...

nighthawk12 said:

Hello,

I have thousands of raw separate files, with over 30 million Instagram emails and 100million Instagram usernames. I used to use Microsoft Access to filter my data. But at the moment it always crashes.
Since I'm going to be scraping about 100k emails - 800k Records per day. I will need to upgrade my database. Because its impossible to run queries atm

Does anyone know how i can make a database, that works fast?

That Filter duplicates and exports unique values and options without being in Microsoft access?


Thank you in advance.

Click to expand...