1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ahrefs crawler is fast as THE FLASH

Discussion in 'BlackHat Lounge' started by darulez, Jul 18, 2017.

  1. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    3,066
    Likes Received:
    1,067
    Gender:
    Male
    Occupation:
    Doing Internet Warfare
    Location:
    Bad Neighborhood
    amazed..

    today I got some outreach link on some "hillbilly" Site - it's clean, it's related, is free. Why not.
    Just a few hours later ahrefs caught it up.

    yeah!
     
  2. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,294
    Likes Received:
    8,088
    It indeed is. They crawl 6 billion pages every day.
    According to this report, it is the 2nd most active web crawler on the internet. 1st is obviously GoogleBot.
     
    • Thanks Thanks x 3
  3. W9go

    W9go Elite Member

    Joined:
    May 16, 2011
    Messages:
    5,205
    Likes Received:
    1,089
    Gender:
    Male
    Occupation:
    chasing girls
    Location:
    chasing girls
    thats quite impressive ... how much does g have ??
     
  4. stack paper

    stack paper Jr. VIP Jr. VIP

    Joined:
    Jan 24, 2008
    Messages:
    617
    Likes Received:
    318
    Occupation:
    Analog Nomad
    Location:
    Reddit / Chiang Mai
    Home Page:
    Ahrefs is the truthhhhh.
     
  5. cosmo89

    cosmo89 Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 25, 2014
    Messages:
    177
    Likes Received:
    33
    Wow!! I used their service as well... Dashboard looks clear and simple... also the live indexing page so much useful.
     
  6. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    3,066
    Likes Received:
    1,067
    Gender:
    Male
    Occupation:
    Doing Internet Warfare
    Location:
    Bad Neighborhood
    just hope that the once do a free check up like majestics or moz does.. without login

    that would leave me with less fuckups to work with to argue at
     
  7. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,751
    Likes Received:
    11,416
    Occupation:
    COINZ
    Location:
    BUYAH
    Home Page:
    I can't believe Bingbot is slower than ahrefsbot. Also I'm pretty sure Googlebot covers a lot more than 3,9% of the WWW. I think the data for that infographic is way skewed. Take this with a grain of salt.
     
  8. virtualpurity

    virtualpurity Jr. VIP Jr. VIP

    Joined:
    Nov 12, 2012
    Messages:
    851
    Likes Received:
    495
    Occupation:
    SEO, Hosting
    Location:
    /root
    Home Page:
    Nonsense, nothing/nobody is faster than The Flash obviously.
     
  9. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    3,066
    Likes Received:
    1,067
    Gender:
    Male
    Occupation:
    Doing Internet Warfare
    Location:
    Bad Neighborhood
    [QUOTE="bartosimpsonio, post: 10234380, member: 653585"]I can't believe Bingbot is slower than ahrefsbot. Also I'm pretty sure Googlebot covers a lot more than 3,9% of the WWW. I think the data for that infographic is way skewed. Take this with a grain of salt.[/QUOTE]

    I can believe that.
     
  10. davids355

    davids355 Jr. VIP Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    10,401
    Likes Received:
    8,105
    That's impressive. Would love to see the kit that ahrefs are using to trawl all that data.
     
  11. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    439
    Likes Received:
    221
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    I'm wondering what language it's written in. I'm going to wild guss it and go with C. Also interesting, perhaps even more so than the language, is the database, are they using BigData like Google?

    On a side note, they crawl and store pretty much the same amount of data as Google but ahrefs must make like 1 percent of what google makes.
     
  12. FORSAK3N

    FORSAK3N Junior Member

    Joined:
    Jun 18, 2017
    Messages:
    127
    Likes Received:
    28
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Los Angeles
    Bing is very slow and borrows off of google ALOT My Yelp review update for Google is about 3 days - Bing is about two weeks. I would say Ahref sorts out backlinks pretty quick for a small company but not content
     
  13. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,751
    Likes Received:
    11,416
    Occupation:
    COINZ
    Location:
    BUYAH
    Home Page:
    OCaml

    They don't crawl anywhere near Google.
     
  14. FORSAK3N

    FORSAK3N Junior Member

    Joined:
    Jun 18, 2017
    Messages:
    127
    Likes Received:
    28
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Los Angeles
    I think they crawl for backlinks at about 30% the speed of google from what I have seen on 5 websites when a new backlink goes up ... where they are REALLY fall behind google is content, meaning keywords, organic search, etc
     
  15. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    439
    Likes Received:
    221
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    Jesus Christ didn't know about that one, it's amazing how many pop up over tge years. And it appears that the mid 90s were hot for programming languages: java, php, ruby, python (a bit earlier) and now this "ocaml".

    I would've thought they use a lower level lamguage due to the intensive data processing they need to do every Day.
     
  16. bonerlicious

    bonerlicious BANNED BANNED

    Joined:
    Apr 12, 2017
    Messages:
    60
    Likes Received:
    84
    Gender:
    Male
    majestic gave me a free lifetime enterprise subscription and for that reason I love them more than ahrefs
     
    • Thanks Thanks x 1
  17. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,751
    Likes Received:
    11,416
    Occupation:
    COINZ
    Location:
    BUYAH
    Home Page:
    Someone in their initial team must be French. The French really love OCaml :p
     
  18. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,294
    Likes Received:
    8,088
    Its 3.9% of the "good" bot traffic.
    And ahrefs bot being faster than bing bot isn't that hard to believe when you see ahrefs picking up links within hours.

    Comparing Ahrefs to Google is like comparing apples to oranges.
    Anyway, ahrefs uses 2 Petabytes or roughly 20 million Gigabytes of storage. Google's index is supported by well over 100 million Gigabytes of storage.
     
  19. roki4ka

    roki4ka Senior Member

    Joined:
    Jun 20, 2016
    Messages:
    888
    Likes Received:
    649
    Strange I find Yandex to cralw faster, just next to googlebot, in various monitoring platforms.
     
  20. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,751
    Likes Received:
    11,416
    Occupation:
    COINZ
    Location:
    BUYAH
    Home Page:
    If these numbers were correct it implies a 4:1 Google:AHrefs crawl ratio. Possible, but I find it hard to believe. About Bing, well it's hard to argue in favor of Bing LOL.