[Journey] Building a free SEO REST API

[Deleted member 1940830]

Hi BHW!

I started to build an SEO API and I want to share my journey here.

First step, the PoC, actually, I retrieve data from several sources (semrush, moz, ubersuggest), it's free and unlimited!

Under the hood, the process leverages reverse engineering and various techniques that I won't disclose here.

Here is a preview :


Next steps:
- Create a documentation.
- Create a website.
- Implement majestic metrics.
- Add an endpoint for keyword metrics (monthly volume, etc...).
- [Your idea here]

 

[Larry Igna]

First step, the PoC, actually, I retrieve data from several sources (semrush, moz, ubersuggest), it's free and unlimited!

Just as annoying as it might seem this question, did I understand well that you can grab relevant data by scraping these? If you reverse engineered them to get only half of data, that's still impressive.
I don't want you to reveal your method, only confirm that you don't use paid accounts for it.

 

[Deleted member 1940830]

Just as annoying as it might seem this question, did I understand well that you can grab relevant data by scraping these? If you reverse engineered them to get only half of data, that's still impressive.
I don't want you to reveal your method, only confirm that you don't use paid accounts for it.

All I can say is that it is not scraping, and it doesn't use paid accounts
The system doesn't get only half of the data, but 100%! The example just shows a small part of the iceberg.

 

[Larry Igna]

Great job then, good luck with your journey, this can be easily monetized.

 

[Deleted member 1940830]

Great job then, good luck with your journey, this can be easily monetized.

Thank you! Monetization is not a problem, the biggest challenge will be promotion/marketing of the tool.
But as it is free, I think it will facilitate the things.

 

[Next Hubz Limited]

Good job & go ahead with your journey

 

[Deleted member 1940830]

Hi BHW, and thanks to those who are following my journey!

Actually, I'm working on the very next step: the documentation.
I tried many solutions, but surprisingly, none of them fits with 100% of my needs.

But, although I hate Facebook/Meta and ReactJS, I ended up choosing Docusaurus for the following key features:
- Modern design.
- Flexibility.
- (truly) Open-source.
- Self-hosted.
Only one thing was missing: an API playground component that allows you to try the API easily, without setting up anything.
And I'm actually building this component by myself.

Here is a preview of this component, using a "fake" API for testing purposes:


Of course, there is a light version:


Once the component will be finished, I'll start to write the documentation itself.

 

[sayan162]

Hi BHW!

I started to build an SEO API and I want to share my journey here.

First step, the PoC, actually, I retrieve data from several sources (semrush, moz, ubersuggest), it's free and unlimited!

Under the hood, the process leverages reverse engineering and various techniques that I won't disclose here.

Here is a preview :
View attachment 348606

Next steps:
- Create a documentation.
- Create a website.
- Implement majestic metrics.
- Add an endpoint for keyword metrics (monthly volume, etc...).
- [Your idea here]

Don't want to discourage you but if you bypassing their security then more or less it will be fixed by them soon. May be you can start reselling seo data from dataforseo, that will be longterm project.

 

[Deleted member 1940830]

Don't want to discourage you but if you bypassing their security then more or less it will be fixed by them soon. May be you can start reselling seo data from dataforseo, that will be longterm project.

It's not really about bypassing securities, it's more about using their system in a roundabout way.
If they want to "fix" it, they need to rework their system and spend so much time/money, additionally, I have fallback methods to prevent this case.

For example, moz DA is available from moz, but also from ubersuggest, and so on.
I have to implement more data sources but there will be multiple methods available for each metrics.

Also, a cache system will be added to avoid a large number of identical requests.

So you didn't discourage me, you encouraged me to work more on this aspect

 

[Deleted member 1940830]

Update

So, I created a landing page.


It's a preview, and it can change in the future, but I like it as it is for now.
Next things to implement here: auth system and user area to generate API key, and the rest of the home page.

On the API, I have to make it more robust, including fallback methods to retrieve data.

I also have to write the documentation, but I wait to finish the definitive API for that.

 

[sayan162]

It's not really about bypassing securities, it's more about using their system in a roundabout way.
If they want to "fix" it, they need to rework their system and spend so much time/money, additionally, I have fallback methods to prevent this case.

For example, moz DA is available from moz, but also from ubersuggest, and so on.
I have to implement more data sources but there will be multiple methods available for each metrics.

Also, a cache system will be added to avoid a large number of identical requests.

So you didn't discourage me, you encouraged me to work more on this aspect

Great that will be looking forward.

 

[Deleted member 1904243]

Great

Hi BHW!

I started to build an SEO API and I want to share my journey here.

First step, the PoC, actually, I retrieve data from several sources (semrush, moz, ubersuggest), it's free and unlimited!

Under the hood, the process leverages reverse engineering and various techniques that I won't disclose here.

Here is a preview :
View attachment 348606

Next steps:
- Create a documentation.
- Create a website.
- Implement majestic metrics.
- Add an endpoint for keyword metrics (monthly volume, etc...).
- [Your idea here]

Great job , Best of luch go ahead with your journey

 

[Deleted member 1940830]

Update:

Today I worked hard on the API, I rebuilt a big part of the system, to make it faster and more sustainable.
I also rearranged the code to make it easier to implement new features, and possibly change the things faster in case of the 'fix' we talked about earlier with sayan162

I still have several things to do here:
- Implement a cache system.
- Implement an additional fallback method to fetch the data.
- Add several options.

Concerning majestic, data is not available like it is with the other tools, and the only way I found will slow down a lot the API, so I will not implement it right now, maybe in the future.

 

[pomki]

Update:

Today I worked hard on the API, I rebuilt a big part of the system, to make it faster and more sustainable.
I also rearranged the code to make it easier to implement new features, and possibly change the things faster in case of the 'fix' we talked about earlier with https://www.blackhatworld.com/goto/post?id=17743990

I still have several things to do here:
- Implement a cache system.
- Implement an additional fallback method to fetch the data.
- Add several options.

Concerning majestic, data is not available like it is with the other tools, and the only way I found will slow down a lot the API, so I will not implement it right now, maybe in the future.

This is amazing! Isn't all the raw data from Majestic. Moz, Ubersuggest etc behind the paid API services, how do you access this for free? Anyways, this project is bomb.

 

[Deleted member 1940830]

This is amazing! Isn't all the raw data from Majestic. Moz, Ubersuggest etc behind the paid API services, how do you access this for free? Anyways, this project is bomb.

Thank you! I will not reveal the manufacturing secrets haha, but all I can say is that the data is publicly available if you understand how the tools works behind the scene, there is also few securities to bypass but nothing really strong (yet?).

Update:
I added a cache system, when the data is cached, response is almost instant

 

[jimbobo2779]

Don't want to discourage you but if you bypassing their security then more or less it will be fixed by them soon. May be you can start reselling seo data from dataforseo, that will be longterm project.

It's probably coming from the SEOToolbar. It works for a while, until you scale at which point he is going to need a fuck load of proxies. Free and unlimited are a pipe dream.

@ OP. Careful with this. If you don't pay these companies and have an agreement in place, they will come knocking. Ask me how I know, lol

 

[IMTherapist]

Hello

Their 3rd party data might be misleading

I think you should use some Google APIs and private methods using niche specific, deeper analysis than some 3rd parties do.

Good luck

For now it looks like an idea put together in a rush. What will it tell you about pbns etc? Nothing. I don't consider 3rd party useful.

What is the journey here? I don't see any. It looks like a hobby project with no goal.

Maybe someone will buy it - but that's what every creator with no goal and validation assumed.

Being dependent on these 3rd party sites I guess there's no long term plan, just dependency on someone's data. Api glue.

 

[Deleted member 1940830]

It's probably coming from the SEOToolbar. It works for a while, until you scale at which point he is going to need a fuck load of proxies. Free and unlimited are a pipe dream.

@ OP. Careful with this. If you don't pay these companies and have an agreement in place, they will come knocking. Ask me how I know, lol

The data come directly from the tools, it doesn't leverage SEOToolbar, or any tools that are already free.
It doesn't need any proxies the way I did it, but it doesn't mean that they can track me anyway.
They can come knocking, it will not secure their system. Even if they want to secure it, they'll need to make big changes in the core, that will be very expensive for them.

Hello

Their 3rd party data might be misleading

They don't provide misleading data, but estimated values, and it can be manipulated, yes.

I think you should use some Google APIs and private methods using niche specific, deeper analysis than some 3rd parties do.

It's not the goal of this project, and what your advice is unrealistic for a free solution.
Google APIs are more accurate, but are also a nightmare to implement, a nightmare to fit the requirements, and can be (very) expensive, especially for big volumes.
Some of their APIs requires a significant spending amount in Gads.

Good luck

For now it looks like an idea put together in a rush. What will it tell you about pbns etc? Nothing. I don't consider 3rd party useful.

What is the journey here? I don't see any. It looks like a hobby project with no goal.

The goal of this thread is to document the path of this project, from start to release, it is what we call a "journey", isn't it?

The goal of this project is to provide a free API that permit to build on top of it, for example, an SEO dashboard to see data from the popular tools for a list of websites at a glance, and save time navigating, logging-in, and searching for this data manually.
Another example: fetch the basic metrics for a list of domains names in a spreadsheet, for link building or backlinks selling.
There is many use cases.

I don't consider 3rd party useful

It's your advice, I suggest you to build your own tool that fit your requirements.

 

[Larry Igna]

@ OP. Careful with this. If you don't pay these companies and have an agreement in place, they will come knocking. Ask me how I know, lol

Being dependent on these 3rd party sites I guess there's no long term plan, just dependency on someone's data. Api glue.

Once this is released and a few hundreds of people will use it at once, all these problems will find their answer, as OP's success or as a big fail.
Either way, this journey is worth watching and I can see OP learning something from this (he seems very knowledgeable as programmer if the first post is true). And I'm definitely learning few things from others journeys, as long as they are updated hehe

 

[jimbobo2779]

The data come directly from the tools, it doesn't leverage SEOToolbar, or any tools that are already free.
It doesn't need any proxies the way I did it, but it doesn't mean that they can track me anyway.
They can come knocking, it will not secure their system. Even if they want to secure it, they'll need to make big changes in the core, that will be very expensive for them.



They don't provide misleading data, but estimated values, and it can be manipulated, yes.


It's not the goal of this project, and what your advice is unrealistic for a free solution.
Google APIs are more accurate, but are also a nightmare to implement, a nightmare to fit the requirements, and can be (very) expensive, especially for big volumes.
Some of their APIs requires a significant spending amount in Gads.


The goal of this thread is to document the path of this project, from start to release, it is what we call a "journey", isn't it?

The goal of this project is to provide a free API that permit to build on top of it, for example, an SEO dashboard to see data from the popular tools for a list of websites at a glance, and save time navigating, logging-in, and searching for this data manually.
Another example: fetch the basic metrics for a list of domains names in a spreadsheet, for link building or backlinks selling.
There is many use cases.

It's your advice, I suggest you to build your own tool that fit your requirements.

So you "reverse engineered" it, yet you say you are getting the stats direct from them. You say you aren't using proxies yet they don't know who you are. This is starting to look really fishy.

I started off thinking you were being naive in thinking you could accomplish this, now I think you are trying to scam people.

You started your account a couple days ago and have come out the gate running saying you are going to provide something to everyone for free and it will be unlimited.

Matey, I spend many thousands every month to provide Moz and Majestic stats, if you think you can sustainably do it without attracting their attention you are delusional.

This seems more like free advertising than a journey.