1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ahrefs crawler is fast as THE FLASH

Discussion in 'BlackHat Lounge' started by darulez, Jul 18, 2017.

  1. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    2,934
    Likes Received:
    1,037
    Gender:
    Male
    Occupation:
    Doing Internet Warfare
    Location:
    Bad Neighborhood
    Home Page:
    amazed..

    today I got some outreach link on some "hillbilly" Site - it's clean, it's related, is free. Why not.
    Just a few hours later ahrefs caught it up.

    yeah!
     
  2. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,226
    Likes Received:
    8,005
    It indeed is. They crawl 6 billion pages every day.
    According to this report, it is the 2nd most active web crawler on the internet. 1st is obviously GoogleBot.
     
    • Thanks Thanks x 3
  3. W9go

    W9go Jr. VIP Jr. VIP Premium Member

    Joined:
    May 16, 2011
    Messages:
    4,861
    Likes Received:
    1,001
    Gender:
    Male
    Occupation:
    chasing girls
    Location:
    chasing girls
    thats quite impressive ... how much does g have ??
     
  4. stack paper

    stack paper Jr. VIP Jr. VIP

    Joined:
    Jan 24, 2008
    Messages:
    585
    Likes Received:
    298
    Occupation:
    Analog Nomad
    Location:
    Reddit / Chiang Mai
    Home Page:
    Ahrefs is the truthhhhh.
     
  5. cosmo89

    cosmo89 Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 25, 2014
    Messages:
    157
    Likes Received:
    33
    Wow!! I used their service as well... Dashboard looks clear and simple... also the live indexing page so much useful.
     
  6. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    2,934
    Likes Received:
    1,037
    Gender:
    Male
    Occupation:
    Doing Internet Warfare
    Location:
    Bad Neighborhood
    Home Page:
    just hope that the once do a free check up like majestics or moz does.. without login

    that would leave me with less fuckups to work with to argue at
     
  7. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,476
    Likes Received:
    11,179
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    I can't believe Bingbot is slower than ahrefsbot. Also I'm pretty sure Googlebot covers a lot more than 3,9% of the WWW. I think the data for that infographic is way skewed. Take this with a grain of salt.
     
  8. virtualpurity

    virtualpurity Jr. VIP Jr. VIP

    Joined:
    Nov 12, 2012
    Messages:
    764
    Likes Received:
    442
    Occupation:
    SEO, Hosting
    Location:
    /root
    Home Page:
    Nonsense, nothing/nobody is faster than The Flash obviously.
     
  9. darulez

    darulez Jr. VIP Jr. VIP

    Joined:
    Mar 12, 2013
    Messages:
    2,934
    Likes Received:
    1,037
    Gender:
    Male
    Occupation:
    Doing Internet Warfare
    Location:
    Bad Neighborhood
    Home Page:
    [QUOTE="bartosimpsonio, post: 10234380, member: 653585"]I can't believe Bingbot is slower than ahrefsbot. Also I'm pretty sure Googlebot covers a lot more than 3,9% of the WWW. I think the data for that infographic is way skewed. Take this with a grain of salt.[/QUOTE]

    I can believe that.
     
  10. davids355

    davids355 Jr. VIP Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    10,182
    Likes Received:
    7,830
    Home Page:
    That's impressive. Would love to see the kit that ahrefs are using to trawl all that data.
     
  11. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    439
    Likes Received:
    220
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    I'm wondering what language it's written in. I'm going to wild guss it and go with C. Also interesting, perhaps even more so than the language, is the database, are they using BigData like Google?

    On a side note, they crawl and store pretty much the same amount of data as Google but ahrefs must make like 1 percent of what google makes.
     
  12. FORSAK3N

    FORSAK3N Junior Member

    Joined:
    Jun 18, 2017
    Messages:
    124
    Likes Received:
    23
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Los Angeles
    Bing is very slow and borrows off of google ALOT My Yelp review update for Google is about 3 days - Bing is about two weeks. I would say Ahref sorts out backlinks pretty quick for a small company but not content
     
  13. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,476
    Likes Received:
    11,179
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    OCaml

    They don't crawl anywhere near Google.
     
  14. FORSAK3N

    FORSAK3N Junior Member

    Joined:
    Jun 18, 2017
    Messages:
    124
    Likes Received:
    23
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Los Angeles
    I think they crawl for backlinks at about 30% the speed of google from what I have seen on 5 websites when a new backlink goes up ... where they are REALLY fall behind google is content, meaning keywords, organic search, etc
     
  15. rafark

    rafark Regular Member

    Joined:
    Jan 15, 2013
    Messages:
    439
    Likes Received:
    220
    Gender:
    Male
    Occupation:
    Moderador
    Location:
    Noble and Heroic MC
    Jesus Christ didn't know about that one, it's amazing how many pop up over tge years. And it appears that the mid 90s were hot for programming languages: java, php, ruby, python (a bit earlier) and now this "ocaml".

    I would've thought they use a lower level lamguage due to the intensive data processing they need to do every Day.
     
  16. bonerlicious

    bonerlicious BANNED BANNED

    Joined:
    Apr 12, 2017
    Messages:
    60
    Likes Received:
    84
    Gender:
    Male
    majestic gave me a free lifetime enterprise subscription and for that reason I love them more than ahrefs
     
    • Thanks Thanks x 1
  17. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,476
    Likes Received:
    11,179
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    Someone in their initial team must be French. The French really love OCaml :p
     
  18. bk071

    bk071 Jr. Executive VIP Jr. VIP

    Joined:
    Nov 24, 2010
    Messages:
    3,226
    Likes Received:
    8,005
    Its 3.9% of the "good" bot traffic.
    And ahrefs bot being faster than bing bot isn't that hard to believe when you see ahrefs picking up links within hours.

    Comparing Ahrefs to Google is like comparing apples to oranges.
    Anyway, ahrefs uses 2 Petabytes or roughly 20 million Gigabytes of storage. Google's index is supported by well over 100 million Gigabytes of storage.
     
  19. roki4ka

    roki4ka Senior Member

    Joined:
    Jun 20, 2016
    Messages:
    887
    Likes Received:
    644
    Strange I find Yandex to cralw faster, just next to googlebot, in various monitoring platforms.
     
  20. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,476
    Likes Received:
    11,179
    Occupation:
    CHEAP
    Location:
    DATASETS
    Home Page:
    If these numbers were correct it implies a 4:1 Google:AHrefs crawl ratio. Possible, but I find it hard to believe. About Bing, well it's hard to argue in favor of Bing LOL.