own ahrefs/majestic seo tool (crawler)
- Thread starter D4rQW4v3
- Start date
I will prioritize what i need. I don't want to lie or misinform I'm a SEO consultant and "holiday programmer" learning day by day - so I guess it will be a server-like app because a simple python script might not be that effective. Yes there is ahrefs api that I should use (with credits), majestic paid api and so on but seriously I know people (not personally) who got their own crawlers and SW that crawled billions of pages and got better analysis. I could even buy such software.
About firehose and such, that's a bigger level that I seriously do not need (for now), also paid and I'm kinda not a big social dataminer.
So here is what I need:
1.Basically I need backlinks fe. I want to get the information about URLs (type of the url: img, redirect, sitewide, nofollow,etc.), I think about it as crawling the whole INTERNET looking for "<a href>" in html that contain the URL right? But it will be more sophisticated because we can't find all the links with google operators like "inurl, site, link etc."
1a. if possible I would like to track new/lost
1b. not needed: (because I can see the linkprofile in masjestic seo) quality of the link (like pr,ct,tf and other factors)
2. I would also like crawler anchor texts
3. ref domains
4. keyword rankings (daily/monthly)
5. social activity
I will be very very glad for your advice and ready to buy you a cup of coffe or tea because I actually am a freelancer and for my projcets it would be very helpful to have such tool (one day i will combine everything and release ultimate tool because now it's a little bit of everything everywhere). I am becoming whitehat, adding value to content, doing longterm seo but I need to watch competitors that's why I need these services.
edit: about the question "imitate the scale of ahrefs/majestic/etc.
I don't really need THAT BIG crawler or goal like that. I'd use it for personal projects or with my team or invite more people but I do not need that to be SO GLOBAL.
If it's possible I would like to for example crawl (if it's easier for the bot) all the "furniture" sites or some regions like German, French, Czech etc. sites. I don't know what's easier (if it matters) to the crawler, that crawler doesn't need to crawl all the bullsh*
About firehose and such, that's a bigger level that I seriously do not need (for now), also paid and I'm kinda not a big social dataminer.
So here is what I need:
1.Basically I need backlinks fe. I want to get the information about URLs (type of the url: img, redirect, sitewide, nofollow,etc.), I think about it as crawling the whole INTERNET looking for "<a href>" in html that contain the URL right? But it will be more sophisticated because we can't find all the links with google operators like "inurl, site, link etc."
1a. if possible I would like to track new/lost
1b. not needed: (because I can see the linkprofile in masjestic seo) quality of the link (like pr,ct,tf and other factors)
2. I would also like crawler anchor texts
3. ref domains
4. keyword rankings (daily/monthly)
5. social activity
I will be very very glad for your advice and ready to buy you a cup of coffe or tea because I actually am a freelancer and for my projcets it would be very helpful to have such tool (one day i will combine everything and release ultimate tool because now it's a little bit of everything everywhere). I am becoming whitehat, adding value to content, doing longterm seo but I need to watch competitors that's why I need these services.
edit: about the question "imitate the scale of ahrefs/majestic/etc.
I don't really need THAT BIG crawler or goal like that. I'd use it for personal projects or with my team or invite more people but I do not need that to be SO GLOBAL.
If it's possible I would like to for example crawl (if it's easier for the bot) all the "furniture" sites or some regions like German, French, Czech etc. sites. I don't know what's easier (if it matters) to the crawler, that crawler doesn't need to crawl all the bullsh*
Last edited: