[Journey] Building a free SEO REST API

[jimbobo2779]

Once this is released and a few hundreds of people will use it at once, all these problems will find their answer, as OP's success or as a big fail.
Either way, this journey is worth watching and I can see OP learning something from this (he seems very knowledgeable as programmer if the first post is true). And I'm definitely learning few things from others journeys, as long as they are updated hehe

Don't get too excited, this does not seem at all realistic. Nothing in his first posts suggests anything about his/her ability to accomplish any of this.

You can't "reverse engineer" Moz, Majestic and SEmrush. They are services made up of tons of servers that constantly crawl and analyse the web. They are huge corporations with millions to spend on server infrastructure.

The cost of a server to simply store the data required to reverse engineer such a thing would cost 10s of thousands of dollars, just to store the database. The server to process the data would cost a similarly insane amount. That is with no redundancy or time cost factored into the act of crawling the entire web. It is foolish to believe a lone ranger could do such a thing as their first project.

This is like someone coming and saying they have reverse engineered WoW and added crafting and it is going to be totally free. Consider how many huge corporations have failed at that and how crazy it is for an individual to claim to have done such a thing.

What is likely is the OP has scraped some data from a third party API that provides these stats. That will get shut down with the quickness when you start getting in the thousands of requests a second. Such a service would be getting hit with Ks of requests a second the instant it was made available.

 

[IMTherapist]

Don't get too excited, this does not seem at all realistic. Nothing in his first posts suggests anything about his/her ability to accomplish any of this.

You can't "reverse engineer" Moz, Majestic and SEmrush. They are services made up of tons of servers that constantly crawl and analyse the web. They are huge corporations with millions to spend on server infrastructure.

The cost of a server to simply store the data required to reverse engineer such a thing would cost 10s of thousands of dollars, just to store the database. The server to process the data would cost a similarly insane amount. That is with no redundancy or time cost factored into the act of crawling the entire web. It is foolish to believe a lone ranger could do such a thing as their first project.

This is like someone coming and saying they have reverse engineered WoW and added crafting and it is going to be totally free. Consider how many huge corporations have failed at that and how crazy it is for an individual to claim to have done such a thing.

What is likely is the OP has scraped some data from a third party API that provides these stats. That will get shut down with the quickness when you start getting in the thousands of requests a second. Such a service would be getting hit with Ks of requests a second the instant it was made available.

What can be a better idea is filtering that data so it serves some purpose.

Why not just make a private WoW server. Well, they quickly go down for some reason.

 

[jimbobo2779]

What can be a better idea is filtering that data so it serves some purpose.

Why not just make a private WoW server. Well, they quickly go down for some reason.

Filtering the data would require mass access to a huge proportion or majority of the data at least.

To reverse engineer these services would require access to their index or access to the entire internet. Now, it could be possible with something like common crawl but the amount of processing and storage required would make this an impossible task to accomplish without at least 100k investment.

I looked into this sort of thing and have considered it in the past. The amount of money required, just to process the data is huge, the amount of fast storage required is in the hundreds of TB and that's without any redundancy.

Providing something that will take multiple full time jobs, from skilled workers, and 100K+ in infrastructure alone, for free with no limits is a pie in the sky idea or a scam. Either way I am sceptical that OK can accomplish this off of some obscure post with little explanation.

Until I see any proof that this is anything close to what OP has described I am assuming this is a crude scam in the making.

Do not "invest" or "donate" to this.

A lot of people here have probably forgotten half of the stuff I have done but I have been here for a long time, I have provided free software to the community and have been delivering on software and services for over 10 years. I am quite well versed ( I run multiple APIs, some of which use the Moz, Majestic, Semrush and Ahrefs APIs) in this very small niche

 

[Deleted member 1940830]

So I'll clarify things.

First, if my account is recent, it doesn't mean that I'm not here since years.
As I said in my presentation, I just want to give back some value to the forum.
For those who doubt about my skills, how can you judge someone you don't know?
I am a developer with 10+ years of experience, I worked on a ton of different projects, I know what I do.

This project is not about recreating a complete infrastructure with crawling, data analysis, etc...
It's about retrieving the data from popular SEO tools in a free way, using their system like any user can do.
Ok, I admit there is a little part of scraping, but it's not what retrieves the data.

If it's free, there is no money to spend, no donation, or anything, so no scam here, I'm just sharing the path of the project, you read or not, you like or not, as you want, it will not stop me.
Maybe paid features will be implemented in the future, but it's not the primary goal of this project, and it will follows a strong validation of the sustainability of my system (in other words: big amount of requests).

In the current state, it's an experiment.
Maybe it will be sustainable in the long term, maybe not, it is what we will see here.

Anyway, it looks like my journey is more attracting hate/doubt than encouragement, this was not the goal of the thread.
If you want proof, you will need to wait a little, I'll launch as soon as it's ready for that, but please stop trying to discourage me, this forum was not created for that.

 

[IMTherapist]

So it's not a journey because there's no goal. Should be in general programming chat section I think.

 

[jimbobo2779]

This project is not about recreating a complete infrastructure with crawling, data analysis, etc...

What do you mean by reverse engineered? You cannot pull the stats and not have them see you unless you are scraping a third party at which point the third party will be the bottleneck and you will need tons of proxies at the very least. You either will need a ton of proxies, a ton of free accounts or pay for it.

It's about retrieving the data from popular SEO tools in a free way, using their system like any user can do.

So you're just scraping from a browser plugin. That will not scale beyond the needs of a single user without a lot of proxies. You're claiming this can be done without proxies, I highly doubt it.

If you are using JS to use the users' IP to grab the stats then there absolutely are limits, the same limits as other such sites that do this. If this is what you are doing then you could just say and I'd be off your back. This is not reverse engineering and it is not no limits, it is possible and usable but only for small batches occasionally.

If it's free, there is no money to spend, so no scam here

In your last post you wrote: "So, I plan to build a free SEO API with cheap premium features." The cheap premium features you mention are the issue. You stated your intention clearly, I don't fault you for wanting to make money from your work but you cannot circumvent the rules of this forum by advertising outside of the marketplace or an advertising agreement.

please stop trying to discourage me

I am not discouraging you from continuing, just be honest / realistic about what you are doing. If you are looking to create a paid for product at the end be honest about it and do it within the forum rules.

Stating that you reverse engineered the services and that you will be offering it for free without limits is an issue. It is over promising that will get people's hopes up and start thinking about possibly handing over cash for something that cannot exist.

 

[Larry Igna]

Don't get too excited, this does not seem at all realistic

To reverse engineer these services would require access to their index or access to the entire internet.

Do not "invest" or "donate" to this.

1. I'm not super excited, because SEO is ranked 4th or probably 5th on my interests lists.

2. To reverse engineer these services it's enough to find an unprotected API endpoint (or find ways to bypass protection). Basically, open the gate and extract stuff when needed, don't take everything inside all at once.

3. OP didn't ask for money, but even if he did, this is no BST and thread would be quickly closed. Anyway, I'm happy to repeat your message:

Do not "invest" or "donate" to this.

This is the most interesting technical part for me also:

So you're just scraping from a browser plugin. That will not scale beyond the needs of a single user without a lot of proxies. You're claiming this can be done without proxies, I highly doubt it.

If you are using JS to use the users' IP to grab the stats then there absolutely are limits, the same limits as other such sites that do this. If this is what you are doing then you could just say and I'd be off your back. This is not reverse engineering and it is not no limits, it is possible and usable but only for small batches occasionally.

I'm curious to know this as well, but hey, @anthoserge is free to keep this info private.

 

[jimbobo2779]

2. To reverse engineer these services it's enough to find an unprotected API endpoint (or find ways to bypass protection). Basically, open the gate and extract stuff when needed, don't take everything inside all at once.

That is not reverse engineering these services though, it also does not provide unfettered access to the data. It will be limited, at the very least.

OP states they have access to all stats but curiously only showed the

I'm curious to know this as well, but hey, @anthoserge is free to keep this info private.

Trust me, I have been running this kind of service on BHW for many years already. I have been providing software that relies on similar services for years before that. I have contracts with many of these services and, without any exaggeration at all, I pay thousands every month for the privilege.

He is either straight up wrong / lying about what is being done here or he is just doing a simple JS scrape from a toolbar endpoint. This is not that hard but has lots of limits, this is most likely.

However, he is stating his intention to provide an API for it which cannot use JS so will require scraping by the server which will require a ton of IPs or a ton of infrastructure.

Had the poster not already stated lofty goals and the intention to charge eventually I wouldn't have the issue but this is an impossible promise of something that I'm not really convinced should be on here.

If the mods don't take it down then I can only assume it is fine. I still stand by my point though, that one or all of the intentions stated will be impossible to achieve. And to state you only have intention to provide for free when previously (today) that they intend to charge suggests an intention to charge.

 

[Deleted member 1940830]

After reading all your messages, repeating that I wouldn't disclose my techniques and yet you keep trying to guess over and over, after being insinuated that I was a liar, after being called a scammer, etc... I keep replying to all your messages, trying to convince you that all I presented is real, anyway...

I'm seriously considering stopping this project, keeping everything to myself, or revealing everything and risking ruining one of the methods, I don't care, I'm a developer, and a noob at SEO, I don't need all this tools after all, and it costs me only 3 or 4 days of work until now.

I just wanted to enjoy this project by sharing it with you, get your impressions, your advice, your ideas, your encouragements (the main goal of this section), I thought it was a good idea but apparently not, all I've received is discouragement, doubt, etc... great...
This forum hasn't accustomed me to that.

Yes, I had planned to implement paid features, which I would have done in accordance with the forum rules, in the marketplace section, by getting a Jr. VIP subscription, etc...
That would have followed this first version of the project, which is completely free, with no registration, no collection of personal data, or anything scammy.

In short, I had done everything by the rules.

This forum is BHW, and in "BlackHatWorld", there is "black hat", I thought I was in exactly the right place for a such project, but I guess I was wrong.

Give me some time to think it over and I'll come back to tell you what I'll do with this project...

 

[damnez]

I'm seriously considering stopping this project

keep us updated bro dont let others discourage you, i would like to see more of this project

 

[IMTherapist]

I read and see that you still haven't set a goal for this.

You can simply make a thread in another subforum and link a thread.

I think in journey we focus more on goals, and in other subforums on our ideas.

If you feel offended, consider that maybe you made a mistake for some reason and we just told you that.

You can continue posting here, but the rule is that if you don't want to reply to someone, don't reply to them.

Your problem might be just over-defense. And that's a problem most people have. So just write less defensive posts.

 

[Deleted member 1940830]

Ok, so, I have decided to stop the project permanently. I am now disgusted with the forum, so I will continue on my own path, away from you.
I now understand the decline in the quality of posts on the forum.

For those who wanted one of the techniques I used, I'll just give you a hint: open your developer console in your browser and inspect the requests. You will then understand the internal API key generation that allows access to the data, this is the initial entry point of the alternative method I implemented in my tool.

It is with these words that my short adventure on BHW comes to an end. Thank you to those who supported me.
To the others, I wish you to receive as much negativity as you have given me; then you will understand the pain it causes.
You do not deserve to obtain my knowledge for free.

 

[IMTherapist]

Ok, so, I have decided to stop the project permanently. I am now disgusted with the forum, so I will continue on my own path, away from you.
I now understand the decline in the quality of posts on the forum.

For those who wanted one of the techniques I used, I'll just give you a hint: open your developer console in your browser and inspect the requests. You will then understand the internal API key generation that allows access to the data, this is the initial entry point of the alternative method I implemented in my tool.

It is with these words that my short adventure on BHW comes to an end. Thank you to those who supported me.
To the others, I wish you to receive as much negativity as you have given me; then you will understand the pain it causes.
You do not deserve to obtain my knowledge for free.

Your post is very negative.

You seem to have an exit in open minded community.

Work on your mental health.

The fact you misunderstood the goal of journey section shouldn't determine your next steps.

Negative feedback is only a lesson.

Learn to cope with pain and failure skilfully.

If you think your knowledge is greater than of someone who makes $20m a year, I doubt it. And so I need a proof in some form.

 

[jimbobo2779]

Ok, so, I have decided to stop the project permanently. I am now disgusted with the forum, so I will continue on my own path, away from you.
I now understand the decline in the quality of posts on the forum.

For those who wanted one of the techniques I used, I'll just give you a hint: open your developer console in your browser and inspect the requests. You will then understand the internal API key generation that allows access to the data, this is the initial entry point of the alternative method I implemented in my tool.

It is with these words that my short adventure on BHW comes to an end. Thank you to those who supported me.
To the others, I wish you to receive as much negativity as you have given me; then you will understand the pain it causes.
You do not deserve to obtain my knowledge for free.

You were saying you were going to do something that I know for a fact is absolutely not at all realistic. I don't hold any ill will towards you, I am assuming you were just being naive rather than malicious.

You don't need to leave or stop work on your thing, just don't tell people you are reverse engineering many multi million dollar companies tools and providing it for free but with premium features and it will be free and unlimited for all. This is simply calling a public API endpoint, this works for short runs in short bursts but cannot be used for lists in the hundreds let alone hundreds of API requests a second. If you put this live you would be getting hundreds of people using up multiple connections very quickly, this requires a proper server.

It's not at all realistic and not at all possible. You said you have your poc, let someone test it for you, you'll see very quickly that it won't scale past a single user. At that point you will see that you are going to need a ton of proxies at the very least.

This is not a simple task, it is something I created around 2016 and even back then the limits were tight and hard to circumvent, even with rotating proxies. Eventually you will get contacted by the services and they will want paying. I am not naive in this, I have been one of the few people offering this exact type of service and am very familiar with the ins and outs of running this sort of thing on a small to medium scale.

 

[Deleted member 1940830]

You were saying you were going to do something that I know for a fact is absolutely not at all realistic. I don't hold any ill will towards you, I am assuming you were just being naive rather than malicious.

You don't need to leave or stop work on your thing, just don't tell people you are reverse engineering many multi million dollar companies tools and providing it for free but with premium features and it will be free and unlimited for all. This is simply calling a public API endpoint, this works for short runs in short bursts but cannot be used for lists in the hundreds let alone hundreds of API requests a second. If you put this live you would be getting hundreds of people using up multiple connections very quickly, this requires a proper server.

It's not at all realistic and not at all possible. You said you have your poc, let someone test it for you, you'll see very quickly that it won't scale past a single user. At that point you will see that you are going to need a ton of proxies at the very least.

This is not a simple task, it is something I created around 2016 and even back then the limits were tight and hard to circumvent, even with rotating proxies. Eventually you will get contacted by the services and they will want paying. I am not naive in this, I have been one of the few people offering this exact type of service and am very familiar with the ins and outs of running this sort of thing on a small to medium scale.

A lot of certainty in your message, a lot of prejudices. Maybe you haven't tried hard enough, or maybe you didn't do things the right way.
Too bad for you, you should have thought out-of-the-box; you would have understood that it is truly realistic, even at large scale.
Anyway, I don't care, I'm a developer, I tried to bring some value to the black hat SEO community, but it was a mistake, lesson learned.
It's time for me to get back to my developer stuff, in developer communities, where my projects are welcomed with respect and kindness.

 

[jimbobo2779]

you would have understood that it is truly realistic, even at large scale.

Bro, no. Just no. Like I said, I had a system similar to what you are proposing here, I have experienced it. Say I'm dumb, that's fine. I have provided free software to this community to over 50k users, totally free and I did not monetise that at all. I have had my software cracked, repackaged and sold as a third party. I have run, and still do, many successful software and services over the last 10+ years.

I am not guessing here. If you are sure this can be done, give a power user here access to it. You say you have it working so that should not be an issue. It doesn't need to be open for anyone. Hell show it running through a list of 1K pages/domains, that is not a huge list and should be no issue if it is scalable.

 

[Larry Igna]

I was a fan of this journey, I hope you change your mind.

For those who wanted one of the techniques I used, I'll just give you a hint: open your developer console in your browser and inspect the requests. You will then understand the internal API key generation that allows access to the data, this is the initial entry point of the alternative method I implemented in my tool.

I did that already, what I can say is that Ubersuggest seemed the most easy one.
What I see as a problem though is exactly what @jimbobo2779 pointed out. If you request data using your server only, it will be very easy to get banned (IP, device signatures, behavior etc).
If you rely on user IP to make the API call, then it's not anymore your API. This is where I might got it wrong and there's a way. Especially the caching part gets me confused. But my programming knowledge is somewhat limited.

 

[IMTherapist]

I was a fan of this journey, I hope you change your mind.

I did that already, what I can say is that Ubersuggest seemed the most easy one.
What I see as a problem though is exactly what @jimbobo2779 pointed out. If you request data using your server only, it will be very easy to get banned (IP, device signatures, behavior etc).
If you rely on user IP to make the API call, then it's not anymore your API. This is where I might got it wrong and there's a way. Especially the caching part gets me confused. But my programming knowledge is somewhat limited.

If you don't know much about programming, well...

He is not doing anything complicated or out of the box here. There's many users on BHW who express a lot more of private findings and achievings.

You just need to separate real ones and experienced from newbies lost in the actions and shillers.

A mindset that programming on its own is powerful is not good. It's like saying a degree makes you someone.

No, it's aboit actions you take and plans.

 

[Larry Igna]

You just need to separate real ones and experienced from newbies lost in the actions and shillers.

You were a newbie until yesterday. I can separate that. This guy posting his journey is a newbie too.
Another separation I can make is that he opened my eyes. Now I know that these platforms can be scraped via API calls, even if for personal limited use. Not bad for a newbie.

Now, your posts had nothing but accusations that this is not a journey and few other negative remarks. But hey,if it helps increasing posts number, ok.
Not useful for me, I can make this separation.

As for @jimbobo2779 , his replies were professional and to the point. He clearly looks like he has the experience, both in tech and the business side.

Done separating stuff, unwatching thread.

 

[git]

A lot of certainty in your message, a lot of prejudices. Maybe you haven't tried hard enough, or maybe you didn't do things the right way.
Too bad for you, you should have thought out-of-the-box; you would have understood that it is truly realistic, even at large scale.
Anyway, I don't care, I'm a developer, I tried to bring some value to the black hat SEO community, but it was a mistake, lesson learned.
It's time for me to get back to my developer stuff, in developer communities, where my projects are welcomed with respect and kindness.

As someone who has found and always does find things exposed to the internet that shouldn't be (unlimited API keys, documents, etc) I can surely say what you're saying isn't impossible.

Especially, if you've found a way to generate new API keys on the fly (which isn't far fetched with the raw amount of bad code/exposed endpoints out there.)

From what it seems, is you've run into a lot of push back from people who sell services in the same industry that you've come into, and you're potentially offering free tools in.

So instead of just "leaving", maybe look into why some people are so against what you are doing, as you could be a potential competitor to some.