1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ahrefs crawler is fast as THE FLASH

Discussion in 'BlackHat Lounge' started by darulez, Jul 18, 2017.

  1. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    2,268
    Likes Received:
    706
    Gender:
    Female
    Occupation:
    Waiting 36 days till I can stick it in
    Location:
    Walhalla
    amazed..

    today I got some outreach link on some "hillbilly" Site - it's clean, it's related, is free. Why not.
    Just a few hours later ahrefs caught it up.

    yeah!
     
  2. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,223
    Likes Received:
    8,005
    It indeed is. They crawl 6 billion pages every day.
    According to this report, it is the 2nd most active web crawler on the internet. 1st is obviously GoogleBot.
     
    • Thanks Thanks x 3
  3. W9go

    W9go Jr. VIP Jr. VIP Premium Member

    Joined:
    May 16, 2011
    Messages:
    4,632
    Likes Received:
    932
    Gender:
    Male
    Occupation:
    chasing girls
    Location:
    chasing girls
    thats quite impressive ... how much does g have ??
     
  4. stack paper

    stack paper Jr. VIP Jr. VIP

    Joined:
    Jan 24, 2008
    Messages:
    530
    Likes Received:
    247
    Occupation:
    Analog Nomad
    Location:
    Reddit / Chiang Mai
    Home Page:
    Ahrefs is the truthhhhh.
     
  5. cosmo89

    cosmo89 Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 25, 2014
    Messages:
    125
    Likes Received:
    31
    Wow!! I used their service as well... Dashboard looks clear and simple... also the live indexing page so much useful.
     
  6. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    2,268
    Likes Received:
    706
    Gender:
    Female
    Occupation:
    Waiting 36 days till I can stick it in
    Location:
    Walhalla
    just hope that the once do a free check up like majestics or moz does.. without login

    that would leave me with less fuckups to work with to argue at
     
  7. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,059
    Likes Received:
    10,833
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    I can't believe Bingbot is slower than ahrefsbot. Also I'm pretty sure Googlebot covers a lot more than 3,9% of the WWW. I think the data for that infographic is way skewed. Take this with a grain of salt.
     
  8. virtualpurity

    virtualpurity Jr. VIP Jr. VIP

    Joined:
    Nov 12, 2012
    Messages:
    722
    Likes Received:
    415
    Occupation:
    SEO, Hosting
    Location:
    /root
    Home Page:
    Nonsense, nothing/nobody is faster than The Flash obviously.
     
  9. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    2,268
    Likes Received:
    706
    Gender:
    Female
    Occupation:
    Waiting 36 days till I can stick it in
    Location:
    Walhalla
    [QUOTE="bartosimpsonio, post: 10234380, member: 653585"]I can't believe Bingbot is slower than ahrefsbot. Also I'm pretty sure Googlebot covers a lot more than 3,9% of the WWW. I think the data for that infographic is way skewed. Take this with a grain of salt.[/QUOTE]

    I can believe that.
     
  10. davids355

    davids355 Jr. VIP Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    9,835
    Likes Received:
    7,445
    Home Page:
    That's impressive. Would love to see the kit that ahrefs are using to trawl all that data.
     
  11. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    408
    Likes Received:
    191
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    I'm wondering what language it's written in. I'm going to wild guss it and go with C. Also interesting, perhaps even more so than the language, is the database, are they using BigData like Google?

    On a side note, they crawl and store pretty much the same amount of data as Google but ahrefs must make like 1 percent of what google makes.
     
  12. FORSAK3N

    FORSAK3N Junior Member

    Joined:
    Jun 18, 2017
    Messages:
    121
    Likes Received:
    19
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Los Angeles
    Bing is very slow and borrows off of google ALOT My Yelp review update for Google is about 3 days - Bing is about two weeks. I would say Ahref sorts out backlinks pretty quick for a small company but not content
     
  13. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,059
    Likes Received:
    10,833
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    OCaml

    They don't crawl anywhere near Google.
     
  14. FORSAK3N

    FORSAK3N Junior Member

    Joined:
    Jun 18, 2017
    Messages:
    121
    Likes Received:
    19
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Los Angeles
    I think they crawl for backlinks at about 30% the speed of google from what I have seen on 5 websites when a new backlink goes up ... where they are REALLY fall behind google is content, meaning keywords, organic search, etc
     
  15. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    408
    Likes Received:
    191
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    Jesus Christ didn't know about that one, it's amazing how many pop up over tge years. And it appears that the mid 90s were hot for programming languages: java, php, ruby, python (a bit earlier) and now this "ocaml".

    I would've thought they use a lower level lamguage due to the intensive data processing they need to do every Day.
     
  16. bonerlicious

    bonerlicious BANNED BANNED

    Joined:
    Apr 12, 2017
    Messages:
    60
    Likes Received:
    84
    Gender:
    Male
    majestic gave me a free lifetime enterprise subscription and for that reason I love them more than ahrefs
     
    • Thanks Thanks x 1
  17. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,059
    Likes Received:
    10,833
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    Someone in their initial team must be French. The French really love OCaml :p
     
  18. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,223
    Likes Received:
    8,005
    Its 3.9% of the "good" bot traffic.
    And ahrefs bot being faster than bing bot isn't that hard to believe when you see ahrefs picking up links within hours.

    Comparing Ahrefs to Google is like comparing apples to oranges.
    Anyway, ahrefs uses 2 Petabytes or roughly 20 million Gigabytes of storage. Google's index is supported by well over 100 million Gigabytes of storage.
     
  19. roki4ka

    roki4ka Senior Member

    Joined:
    Jun 20, 2016
    Messages:
    869
    Likes Received:
    636
    Strange I find Yandex to cralw faster, just next to googlebot, in various monitoring platforms.
     
  20. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,059
    Likes Received:
    10,833
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    If these numbers were correct it implies a 4:1 Google:AHrefs crawl ratio. Possible, but I find it hard to believe. About Bing, well it's hard to argue in favor of Bing LOL.