1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Ahrefs vs. Majestic RDs

Discussion in 'Black Hat SEO' started by wizeman, Sep 26, 2019.

  1. AngelSeo

    AngelSeo Jr. VIP Jr. VIP

    Joined:
    Nov 3, 2017
    Messages:
    1,589
    Likes Received:
    878
    Gender:
    Male
    LooL this guy and his script. I blocked him sometimes ago after such pointless argument and now was like WTF to whom those people talk. Yeah you can safely ignore whatever he says - he have no clue whatsoever.
     
  2. Purush

    Purush Jr. VIP Jr. VIP

    Joined:
    Jul 12, 2016
    Messages:
    3,506
    Likes Received:
    811
    Gender:
    Male
    Home Page:
    Then y r u here. Teasing others ?
     
  3. FatBee

    FatBee Jr. VIP Jr. VIP

    Joined:
    Apr 4, 2015
    Messages:
    1,213
    Likes Received:
    601
    Gender:
    Male
    i award you with Rank #1 Internet Warrior.

    you run out of content so you start digging old posts, so you can better understand me?

    Does google evolved since 2015? Yes.

    Is there any public info about what i did since than? No.
     
  4. GregFromMoonsy

    GregFromMoonsy Power Member

    Joined:
    Aug 12, 2018
    Messages:
    548
    Likes Received:
    195
    Gender:
    Male
    Location:
    European circus
    @Stas Va

    Good that you know the difference between historic and fresh index.
    For any new site which is under 3 months in age - historic index is completely useless - I am agree here.

    For any older site you don't see in fresh index backlinks which was discovered 4 months ago and not crawled again in last 3 months.
    Such situations are the most common, as the result you not see significantly part of backlinks in fresh index.

    Because Majestic have 50x larger pages index than Ahrefs, there is not possible to crawl all those pages in 3 months.
    For this reason Ahrefs have more backlinks in fresh index, but much less backlinks when we will sum fresh and historic index.

    And this is why professionals use historic index and choose Majestic.


    Let's we talk now about crawling system.

    By poor written scripts you aren't able to effective crawl the web.
    I'll explain you how working my system, it will be easier to you understand the point.

    So, before any single host/page are crawled - for all the hosts system will get the IP for each one.
    As the next step base on IP/urls - queue being created in the way, that it not will overwhelm any single server/host.

    In crawling process, information about single host speed / http code - modifying the queue.
    For example, when the host replying slowly - the next request is made after longer time, same as doing it Google.

    Also when some host trying to block my crawlers - some other algorithms are used (other server location, proxy,etc)
    Mentioned recalcitrant host is crawled until successful crawl, then used successful method is stored for future use.

    So, there no any chance to hide anything as some amateurs think - in reality they're not able to block nothing :D
    In that simple way my system solving all crawling problems.

    However, due to multi server application - it's not easy to create it.
    Because it need to modify the queue "in the air" on multi servers in multi locations.

    Cheers, Greg.
     
    • Thanks Thanks x 1
    Last edited: Oct 1, 2019
  5. Stas Va

    Stas Va Newbie

    Joined:
    Oct 1, 2019
    Messages:
    34
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Hacking bots and human behavior

    1. Again ! my accusation to Majestic - was not the size of the database. Relevancy is not about size. Database is updating not good enough (comparing with ahrefs and Semrush). Historical Index should be use only to understand backlink profile changes. It technically can't be relevant.


    2. No sense. Servers are crushing because of to much queries to the server in a short period of time. Server and site scripts of the referral can't handle it and crushes (usually 503). No difference what IP and how many IP you will use, from what servers or devises/os/browser/location you will try to make requests. I am not talking about any type of blocking or captcha. So please don't change the context.
    So can you explain what is the way when you are


    So that's already says that you can't crawl thousands of backlinks in minute without loosing in data. Even if you will create a smart queue of requests, and will manage to check links from the same domain in distance proportional to each other. That won't resolve your problem with the performance that you declare.
    Google-bots are working completely different.

    Wait a minute. Google, ahrefs, Majestic and Semrush - are crawling the web. Not you. That's a completely different approach and concept. Your task as a SEO expert/marketer/employe is to crawl a list of backlinks and to audit them. For example (web tools . de) (sorry can't add links yet) is your client.

    You need to check backlinks to their domain. You can use data from Google Search Console (if you have access) or get data using API from Majestic, ahrefs, Semrush or other data providers. Let's say we are using your favourite Majestic.

    Lets' check overview:
    imgur . com / a / j U B G M q p - great citation flow vs trust flow. 96569 k backlinks and just 20 referrals (fresh index) and 1000000 backlinks in (Historic index)

    So your script will check 90000 backlinks from 20 referrals and you declare your script will do it in minutes, without loosing data. So let's check referrals.

    imgur . com / f 1 N Q t k Z

    What a surprise, 96456 are from 1 domain (99,89 % of referrals) so your script will make a query every few miliseconds to this domain. How you will create querue in this case ? Even if there was 20 % from the same domain, you will need serious time holds to avoid cases when your script crush the site.


    Additional: Back to Majestic DATABASE disput.
    Great notice for you will be that this referral is dead for a long time period. But not for Majestic fresh index. That what I mean when am telling that Majestic data is less relevant. And such error affects all majestic metrics...
     
  6. GregFromMoonsy

    GregFromMoonsy Power Member

    Joined:
    Aug 12, 2018
    Messages:
    548
    Likes Received:
    195
    Gender:
    Male
    Location:
    European circus
    @Stas Va

    1) Check this thread, then you'll see how poor is Ahrefs Fresh index.
    https://www.blackhatworld.com/seo/h...ed-1-without-an-h1-tag.1166662/#post-12492942

    2) If you know how to create and update in air the quene (what I've explained to you) nothing will be crushes.
    Seems, you not understood what I wrote. Because I not wrote about your IP but crawled host IP.
    Also scripting language is not the way to write such effective crawlers.

    3) The point is to crawl thousands IP's at the same time, not the thousands urls at the same host and by this way, noting not will be crashed.
    I am write this from my expirience not any theories.

    4) In partial true. You are not able to index thousands of urls from the single poor host in short time. (see point 2,3)
    I am don't care about poor links from low TF domains, because they have tiny power.
    Especially as you wrote, from the same domain.

    Cheers, Greg.
     
    Last edited: Oct 2, 2019
  7. Ely

    Ely Registered Member

    Joined:
    May 19, 2019
    Messages:
    60
    Likes Received:
    17
    Gender:
    Male
    Hei @GregFromMoonsy , thanks! Tried Majestic (used ahrefs for a long time) today and found a lot of hidden backlinks from my competitors. It's amazing! Thanks again dude!
     
  8. Stas Va

    Stas Va Newbie

    Joined:
    Oct 1, 2019
    Messages:
    34
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Hacking bots and human behavior
    1.Just numbers, same issue you made giving numbers about database size ignoring data relevance.
    Garbage data - useless data. Same arguments, I already destroyed with exact case.

    Can you explain what's info about backlink you will get crawling IP ? How your crawler will eve check if the link is really there or ahrefs attributes ?

    Are u sure now ? Because few posts earlier you told me ...

    And that's the punch line.
     
  9. GregFromMoonsy

    GregFromMoonsy Power Member

    Joined:
    Aug 12, 2018
    Messages:
    548
    Likes Received:
    195
    Gender:
    Male
    Location:
    European circus
    Thank you Ely.
    I am glad that you discovered Majestic to better analyze competitors than with Ahrefs.

    As the next step you should start build the links with right Topic, learn more here:
    https://www.blackhatworld.com/seo/ahrefs-vs-majestic-rds.1165105/page-3#post-12488647


    @Stas Va
    At first - you should throw your team away, because they aren't able to use the brain at all.
    As the next step, you must post the request in "Hire a Freelancer" section with the request to create mentioned solution.

    I am pretty sure that each clever programmer not will be have any hassle with understand what I have on mind with mentioned sentences.
    Especially if he have the real experience in web crawling.


    Let's we talk about the next thing related to Majestic and Ahrefs index:
    Tell me the domain name which have as you said the "garbage data" in Majestic History Index.
    I will destroy your fake story in few minutes, because I am pretty sure that you created your account yesturday only for trolling reason with Ahrefs.

    Seems @AngelSeo also only trolling in this thread. Maybe your nick it's just his second account on BHW.
    He know everything but not understand absolutely nothing same as you.

    As I previously said, Majestic crashing Ahrefs many times.
    Typical example with the fake theory about bigger Ahrefs fresh index:

    example014.png
    https://www.blackhatworld.com/seo/h...ed-1-without-an-h1-tag.1166662/#post-12492942


    So I am not surprised by @Ely post:
    Cheers, Greg.
     
    • Thanks Thanks x 1
    Last edited: Oct 3, 2019
  10. b1step-ahead

    b1step-ahead Supreme Member

    Joined:
    Nov 15, 2010
    Messages:
    1,341
    Likes Received:
    636
    Just get both and you will be good
     
  11. HustleTong

    HustleTong Jr. VIP Jr. VIP

    Joined:
    May 30, 2019
    Messages:
    1,064
    Likes Received:
    238
    Gender:
    Male
    Occupation:
    IM
    Location:
    Old Town Road
    Ahrefs is much faster and gives more professional overview.
     
  12. Mr Positive

    Mr Positive Jr. VIP Jr. VIP

    Joined:
    Mar 30, 2018
    Messages:
    1,376
    Likes Received:
    496
    Gender:
    Male
    Home Page:
    Ahrefs is good but I like moz too both can give deep analysis and are helpful when you’re trying to dig deep
    Into competitors
     
  13. Stas Va

    Stas Va Newbie

    Joined:
    Oct 1, 2019
    Messages:
    34
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Hacking bots and human behavior
    You didn't answer. There is a CLEAR QUESTION ? Who cares what I should say to my team ? What data your crawling system get by Host IP. How your system check if the link even exists on the page, without crawling page ?
    Please give me an example what data your tool is crawling. Can you add a screen ?!

    I will choose site you used earlier.

    Historical DATA:
    Let's check 20 biggest referrals from Majestic.

    [​IMG]

    Oh... checking only first 23 referrals we found out that 318443 - does not exist already on the site with 1 mln backlinks in historical. Lets BET I will find at least 500000 backlinks that doesn't exist. Whats their data size says about that.

    But OK. Everybody knows that for exactly our goal (to check your backlinks) historical index - is useless.
    Let's check referrals from fresh index.

    [​IMG]

    - 2532 (> 7 %) backlinks just after first look
    If you will continue to speculate with Majestic database size and their numbers. I will check all backlinks from this website.
    Because my crawling system really exist, and I can check them all. And I am sure that I will find at least 20 % of Majestic Fresh Database, doesn't works and exists anymore. And doesn't exist for a time period much more then 3-4 months.

    That's absolutely don't boder me. How authoritative opinion of the man who just used Majestic for the first time.

    I do use Majestic, and I also use ahrefs. As I mentioned earlier Majestic do have interesting metrics like TF & CF, that gives you opportunity to understand the ration and so on.
    But only people who didn't work enough with both tools, can deny, that ahrefs database is much cleaner and relevant. Ahrefs & Semrush are great tools for any lvl.
     
    Last edited: Oct 3, 2019
  14. GregFromMoonsy

    GregFromMoonsy Power Member

    Joined:
    Aug 12, 2018
    Messages:
    548
    Likes Received:
    195
    Gender:
    Male
    Location:
    European circus
    I told you, if they (your team) aren't able to understand for which reason they need to check IP when we talking about creating the quene, they don't use the brain at all, because here I haven't on mind "crawling the IP". For this reason - my suggestion to you was: hire the clever programmer who don't will be have any hassle with understand this part.

    Only what is useless here - it's your (and your "team") very low knowledge about crawling the web.
    And bellow you'll find why.

    At first and most important: if you want to crawl a chinese sites (on chinese DNS and/or hosted in China) you should have for this task crawling servers located in mainland China - this is why I am said that for crawling purpose you need servers in many locations.

    It's because they blocking a lot of traffic outside the China at three levels - starting at great China firewall, dns and servers level.
    In this case - all your theories and numbers (related to example with Majestic screenshot) are absolutely worth nothing.

    So please show me the example with Majestic Historical index and many dead links for any domain which haven't links from China.
    Then we can continue to talk.

    Cheers, Greg.
     
    Last edited: Oct 4, 2019
  15. Stas Va

    Stas Va Newbie

    Joined:
    Oct 1, 2019
    Messages:
    34
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Hacking bots and human behavior
    Everything start on the point: "Usually this process (crawling backlinks pages) crush lots of referrals on cheap servers.


    And my question was how your quene allows you not to create request to the site with performance and quality that you have declared. You didn't answer it and I was using your IP quene to troll you. That's don't fix the problem. And to create smart quene of requests and filter them by domains, subnets and so on, was like obvious. On and real site example I demonstrated that's with the perfomance you declared, almost for any average domain you will crush few sites checking backlinks. And actually not always the sites (in most cases, but not always) will be total crap, domains can be good. For example domains with redirects can set on a crappy free hosting and so on. Cases can be very different. You initially declared that GREAT DEVS an your GREAT SCRIPT don't have such problems.

    So me and my team wasn't right ? What you mean saying "partial" true ?
    You have solution ?. Smart Query is obvious "partially" solution, because it's don't fix it fully.
    It can reduce risks using more efficient and uniform load distribution. But not fix, the problem. In most cases >50 % of all backlinks are from 10 % of referrals.
    So that's a very frequent issue.

    Let's go Majestic Historical Data now:
    Let's talk simple local site, that is ranking by "flowers delivery" in miami - https://www.kingscreekflowersandgifts.com/

    [​IMG]

    To make it fair I filtered all deleted backlinks (but actually they are considered in statistic you use).
    If you will delete few duplicats , you will see only.

    200 backlinks 66,4 % doesn't exist, and you get page 404.

    [​IMG]

    BUT LET's CHECK IF BACKLINKS TO https://www.kingscreekflowersandgifts.com/ REALY ARE SET ON REMAINED PAGES.

    [​IMG]


    Screen Shot 2019-10-04 at 3.47.04 PM.png


    You can check all backlink status manually.
    DETAILS FOR ALL BACKLINKS: https://docs.google.com/spreadsheets/d/1OrkMbwhpl1nyphj7L8z7JvOY4saDfEqxtFPPHZ4Zpj4/edit#gid=0
     
  16. GregFromMoonsy

    GregFromMoonsy Power Member

    Joined:
    Aug 12, 2018
    Messages:
    548
    Likes Received:
    195
    Gender:
    Male
    Location:
    European circus
    I told you (but you still can't understand), first you need the IP address for each host to create mentioned queue.
    Without the IP address you aren't able to create the effective queue because:
    1. You don't know which domains are on the same IP or /29 range - in result you crashing single server or you receiving 403 50x http code.
    2. You don't know in which country server is located, in result crawling is slow or even impossible (such mentioned China)
    3. You trying to crawl the domains which even don't have the DNS A,AAAA,CNAME records or domain was deleted, in result crawling have very low effectivity.
    4. You must have daily updated domains database with current domain status (such active,expired,deleted,redemption etc), because in other case - you crawling suspended pages etc (where in result you're not able to find the link)
    As I told you too - quene is modificated in the air - basing on crawling stats, such http code / page speed.
    If you receiving 403 50x http code - you should run additional algorithms to detect if IP was banned or just server have some issues.

    So, you already learned from me a lot of things without even simple "thanks", which cost you nothing.
    Instead of this, you trolling with quoting my sentences, which in reality you aren't able to understand technically.

    What I already learned from you ?
    Technically nothing, except that some people are not worth any help because they already know everything but don't understand nothing.

    Cheers, Greg.
     
    • Thanks Thanks x 1
    Last edited: Oct 6, 2019
  17. Stas Va

    Stas Va Newbie

    Joined:
    Oct 1, 2019
    Messages:
    34
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Hacking bots and human behavior
    Great that our other dispute about the senselessness of the statistics of crawled pages, which "majestic" declare and you have so actively used in this thread was closed.

    3 from 4 of your points are not relevant to our discussion. Our disput is related only with the first point. I already showed you that most of backlinks for almost every average site, are from 10 % of referrals domains. You already acknowledged that queue will not resolve the original problem .

    If we will keep in mind the performance we are talking about.

    So "PRO" why do you steel publish irrelevant commonplace technical things how to create queries on factors that are not related to our problem, if that don't resolve the initial problem. I thought it's not the problem at all.

    How DOMAIN DNS status matters, if we are talking about working domains with sites whose server crashes upon requests ... Which seriously limits either the performance or the quality and quantity of data. You wrote so much, but you wrote nothing.

    :D:weep::D:weep::D

    I'm trolling you, because it funny to see how you try to look more then you are. That's guru role you are trying to play is hilarious. Thanks.

    ps.
    You can destroy me any time, just by showing how "YOUR CRAWLING SYSTEM" in work and report/excell/csv what exactly it crawls. I think that will be an amazing advertisement of your services/tools and so on.
     
  18. GregFromMoonsy

    GregFromMoonsy Power Member

    Joined:
    Aug 12, 2018
    Messages:
    548
    Likes Received:
    195
    Gender:
    Male
    Location:
    European circus
    Ask Ahrefs how many servers they have in China with their "biggest" index :D
    I'll tell you - they using only OVH, AFAIK OVH don't have any single server in China and each amateur is able to block Ahrefs crawlers at iptables level in invisible way.
    Please show me the screenshot of Ahrefs Historical Index + Google sheet with dead BL for this domain, I guess I'll be laugh strong.
    That will be enough to show reality.

    Hope that your amazing crawling script not need the whole week for crawl those few urls from Ahrefs.
    As it was designed by team of first class engineers who asking on forum about help :D

    Cheers, Greg.
     
    Last edited: Oct 7, 2019
  19. Stas Va

    Stas Va Newbie

    Joined:
    Oct 1, 2019
    Messages:
    34
    Likes Received:
    8
    Gender:
    Male
    Occupation:
    Hacking bots and human behavior
    Sure. The result will be less cleaner, because I actually can't filter deleted/removed backlinks backlinks as we did in Majestic (weird, that's there no so option). But not a problem, the most important is the final report. Who will have more active backlinks that really exists.

    [​IMG]

    Moreover to be fair, almost 600 backlinks from 1598 - where duplicates. So overall there are only 977 unique URL's.
    Almost a half are 404 (but keep in mind, we can't filter deleted links, how we made in Majestic).

    [​IMG]

    And the final result is

    [​IMG]


    So what we have ...
    90 (AHREFS) vs 61 (MAJESTIC)
    (32 % difference).
    AHREFS WIN

    All the details: https://docs.google.com/spreadsheets/d/1QfjHg5pnSimOH3p2_mMph4ULkhAfn3uk0rrK68yvJVo/edit?usp=sharing

    Still want to argue with me ? So let's then check another website. But now, you will provide proofs. It should be kinda easy for your script... :smirk:

    1. I didn't ask help. I was wondering how exactly you resolved exact problem, because you declared you resolved it. But after few irrelevant posts you just accepted, that the issue can't be fully fixed.

    Please don't blame me on this. I new here, so can't differ professionals from talkings heads just by nickname yet.

    2. I steel don't see any sign of the existence of your puffy script. After all, it’s not me who spoke in full about his "script" before even somebody ask about it. You just decided that it would be cool to say about a cool technical thing that you don’t have.


    Also can you please tell me, do you feel difference between "pages in index" and "url found"...
    Do you understand the concept behind URL ?


    For example:
    http://site.com/page site.com/page
    https://site.com/page?utm_content=buffercf3b2&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer
    https://site.com/page?utm_content=bullshit21&utm_medium=social&utm_source=linkedin.com&utm_campaign=GregIsWrong
    https://site.com/page

    Are those pages different, and what you will tell about URLs ?

    Please share your "professional" experience...
    What you can tell about internal links, .... do they also have URLs ?

    WHY I'M ASKING ? Because it's interesting why "professional" like you compares two absolute different concepts?

    [​IMG]

    You even didn't use statics about "crawled" URLs. You decided to choose the biggest number in Majestic statistics, and compare with the lowest in Ahrefs statistic (except domains).
    Do you see the difference between "crawled" and "found" ? Great technique for slippy salesman.

    [​IMG]

    I think fake Mythbusters for our industry is way much more dangerous than the myths themselves.
    Professional ? Stop trying to seem like, just be one.
     
    Last edited: Oct 8, 2019 at 5:08 PM
  20. GregFromMoonsy

    GregFromMoonsy Power Member

    Joined:
    Aug 12, 2018
    Messages:
    548
    Likes Received:
    195
    Gender:
    Male
    Location:
    European circus
    @Stas Va

    I see on your Ahrefs screenshot 1,598 backlinks, but in Google sheet you listed ONLY 978
    Can you update it please to full list ?
    I don't like such manipulations.

    When you'll update the Google sheet with Ahrefs BL - I'll run my crawlers to verify all BL.
    Seems, your Ahrefs have bigger chance to win, because Majestic have "only" 1,269 backlinks.

    I'll check the raw BL data without any filter, as you aren't able to use such filter in Ahrefs.
    And I don't like such manipulations as I said previously.

    Then we'll compare my results with yours, to see the reality.
    Hope that you will not disappear in mean time :)

    Just be the man.
    And we'll continue.

    Cheers, Greg.
     
    Last edited: Oct 9, 2019 at 4:51 AM