Instagram Database with over 100million records. How to store all that data?

nighthawk12

nighthawk12

Registered Member
Joined
Oct 3, 2019
Messages
52
Reaction score
38
Hello,

I have thousands of raw separate files, with over 30 million Instagram emails and 100million Instagram usernames. I used to use Microsoft Access to filter my data. But at the moment it always crashes.
Since I'm going to be scraping about 100k emails - 800k Records per day. I will need to upgrade my database. Because its impossible to run queries atm

Does anyone know how i can make a database, that works fast?

That Filter duplicates and exports unique values and options without being in Microsoft access?


Thank you in advance.
 
Advertise on BHW
export the sql file and upload it in to google sheets, there are several ways to create and manage database in google sheets, also is free and fast
 
i think the first question is what are you planning on doing with them? i would filter out the lines that don't match your purposes to cut the size down, then go from there.
 
nighthawk12 said:
Hello,
I have thousands of raw separate files, with over 30 million Instagram emails and 100million Instagram usernames. I used to use
Click to expand...
dont insta members hide there emails ?
seems a bit of a security issue there.
 
proxygo said:
dont insta members hide there emails ?
seems a bit of a security issue there.
Click to expand...
It's hidden by default but on ios/android app users can set up their phone number or e-mail address to be visible for public view on the profile page.

When people set this up, you can scrape their phone/email via mobile API.
 
German Rodriguez said:
export the sql file and upload it in to google sheets, there are several ways to create and manage database in google sheets, also is free and fast
Click to expand...
Hey man, thanks for the suggestion i will start looking into to it!
I was thinking of using an oracle cloud database, they told me is fast and I can filter it as i want.
 
German Rodriguez said:
quick questions are those emails verified?
Click to expand...
Yes, most of my emails are verified. But what I do is when I use them for my needs, I always verify the batch.
To make audiences, cold emails. bulk emails etc.

It depends because I don't use all of them actually, only the ones that meet my requirements.
 
nohupt said:
i think the first question is what are you planning on doing with them? i would filter out the lines that don't match your purposes to cut the size down, then go from there.
Click to expand...
You are correct, I use them for many things building audiences for ads, cold emails, bulk emails. Really depends. The reason why i want to save them all is that its faster since that Instagram data is Gold if you use it correctly, don't want to confuse all the files.
My process is getting the raw file, saving it in the main database, running a query to remove duplicates, save only unique values. Then extract the unique categories(niches) I need.

But i was speaking with an expert, probably going to make an Oracle database. Fast, simple and can access from everywhere since it's in the cloud. Better than local xD
 
proxygo said:
dont insta members hide there emails ?
seems a bit of a security issue there.
Click to expand...

No, I only get the public information that is either in the Biography or the " contact buttons". Where people place to be contacted. Hope this answers your question
 
You can use EmEditor, it's a software that will allows you to load and filter huge files.

I've used it to open and filter huge json files.

The software is paid but you can use the trial version with all it's features, if your license expire just uninstall and install again with another trial account.

Alternatively, if you know some Python. There are some libraries that you can use to filter, extract and segment your data.
 
Last edited:
Algo said:
You can use EmEditor, it's a software that will allows you to load and filter huge files.

I've used it to open and filter huge json files.

The software is paid but you can use the trial version with features, if your license expire just uninstall and install again with another trial account.

Alternatively, if you know some Python. There are some libraries that you can use to filter, extract and segment your data.
Click to expand...
Hey man, thanks for the suggestion. Will for sure check it out.

Yeah i understand a little python since i was helping the dev build my tool, but I'm not an expert at all. :/ Still learning.
 
proxygo said:
dont insta members hide there emails ?
seems a bit of a security issue there.
Click to expand...
neah, it's on plain sight. If you are a business you want your email out there. The login emails are not visible
 
MR.Spuf said:
neah, it's on plain sight. If you are a business you want your email out there. The login emails are not visible
Click to expand...
Exactly, its all good. If they place it is because they want to be contacted xD
 
nighthawk12 said:
Hello,

I have thousands of raw separate files, with over 30 million Instagram emails and 100million Instagram usernames. I used to use Microsoft Access to filter my data. But at the moment it always crashes.
Since I'm going to be scraping about 100k emails - 800k Records per day. I will need to upgrade my database. Because its impossible to run queries atm

Does anyone know how i can make a database, that works fast?

That Filter duplicates and exports unique values and options without being in Microsoft access?


Thank you in advance.
Click to expand...


Mongodb or any file system based DB will help
convert and import the data into it and later access via DB tools
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top