Software/Service Ideas

davids355 · Feb 16, 2018

Not meaning to hijack the thread but I have to say one more thing about ahrefs and it’s cost;

Apparently their system runs on a 20PB storage cluster and 3000 Linux servers so I guess that’s part of the reason why it’s expensive

bhavik85 · Feb 16, 2018

How people make this mass proxy system ( Backconnect proxy )? I am very much interested to own a system like that but I could not find the source of such thing....

rogelio_sevilla1 · Feb 17, 2018

davids355 said:
Not meaning to hijack the thread but I have to say one more thing about ahrefs and it’s cost;

Apparently their system runs on a 20PB storage cluster and 3000 Linux servers so I guess that’s part of the reason why it’s expensive

daaamn 20PB (~20,000,000 GB ) ... AWS Dynamo charges $0.25usd per gigabyte per month of storage. If they are using AWS, they would be paying $0.25 X 20,000,000 (GB) => $5,000,000 usd per month (only for storage). If they're using AWS, I'm pretty sure they got some kind of enterprise deal.

bhavik85 said:
How people make this mass proxy system ( Backconnect proxy )? I am very much interested to own a system like that but I could not find the source of such thing....

Not entirely sure either. As far as I know, AWS assigns one public IP from a pool of reserved IP's to each EC2 instance they provide. I suspect that the companies that provide this kind of service, take advantage of this feature and create a couple of thousands of micro EC2 instances, and somehow, relay a request from X source to any of these available instances. If any of them were to report that it cannot access certain services anymore (let's say, it gets blocked by Google), the EC2 instance would get killed by a monitor process and create a new one with a new IP.

That's just a guess though, I'm far from being a network expert.

rogelio_sevilla1 · Feb 17, 2018

I want to thank all of you for your amazing feedback, I got a couple of ideas from reading what you posted in here.

I will need to do some reading/researching before making sure I can actually do them though, so I guess I will be spending some quality time in Udemy before starting the code work.

Thanks again to everyone!

davids355 · Feb 17, 2018

rogelio_sevilla1 said:
daaamn 20PB (~20,000,000 GB ) ... AWS Dynamo charges $0.25usd per gigabyte per month of storage. If they are using AWS, they would be paying $0.25 X 20,000,000 (GB) => $5,000,000 usd per month (only for storage). If they're using AWS, I'm pretty sure they got some kind of enterprise deal.

Not entirely sure either. As far as I know, AWS assigns one public IP from a pool of reserved IP's to each EC2 instance they provide. I suspect that the companies that provide this kind of service, take advantage of this feature and create a couple of thousands of micro EC2 instances, and somehow, relay a request from X source to any of these available instances. If any of them were to report that it cannot access certain services anymore (let's say, it gets blocked by Google), the EC2 instance would get killed by a monitor process and create a new one with a new IP.

That's just a guess though, I'm far from being a network expert.

I am not sure but possibly they run their own servers. I think AWS is great for startups and basically anyone not really in the tech niche because it makes all aspects of infrastructure management very easy; BUT you pay a price for that. If you are in the tech industry (like ahrefs are) and you know how to run your own hardware then I know you can do the same thing for easily a 10th of the cost. But even so, pretty impressive setup and yea, its going to be very expensive no matter what way they do it.

Software/Service Ideas

davids355

Super Moderator

bhavik85

Regular Member

rogelio_sevilla1

Newbie

rogelio_sevilla1

Newbie

davids355

Super Moderator

Main Menu

Marketplace

Making Money

BlackHat World