1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

own ahrefs/majestic seo tool (crawler)

Discussion in 'General Programming Chat' started by D4rQW4v3, Oct 20, 2013.

  1. D4rQW4v3

    D4rQW4v3 Newbie

    Joined:
    Sep 8, 2010
    Messages:
    18
    Likes Received:
    2
    I would like to create my own tool or buy one. Could you recommend me some good sources? I was thinking about groupbuy for these services but nah I need my own crawler.
     
  2. davids355

    davids355 Jr. VIP Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    10,011
    Likes Received:
    7,702
    Home Page:
    For what purpose exactly, and what's the scope - ie are you trying to imitate the scale of ahrefs/majestic ?
     
  3. john1444

    john1444 Elite Member

    Joined:
    Mar 27, 2012
    Messages:
    2,569
    Likes Received:
    757
    Gender:
    Male
    Occupation:
    Marketer
    Location:
    Miami, FL
    You gotta be specific or we will give you the disavow tool prescription.
     
  4. D4rQW4v3

    D4rQW4v3 Newbie

    Joined:
    Sep 8, 2010
    Messages:
    18
    Likes Received:
    2
    I will prioritize what i need. I don't want to lie or misinform I'm a SEO consultant and "holiday programmer" learning day by day - so I guess it will be a server-like app because a simple python script might not be that effective. Yes there is ahrefs api that I should use (with credits), majestic paid api and so on but seriously I know people (not personally) who got their own crawlers and SW that crawled billions of pages and got better analysis. I could even buy such software.

    About firehose and such, that's a bigger level that I seriously do not need (for now), also paid and I'm kinda not a big social dataminer.

    So here is what I need:


    1.Basically I need backlinks fe. I want to get the information about URLs (type of the url: img, redirect, sitewide, nofollow,etc.), I think about it as crawling the whole INTERNET looking for "<a href>" in html that contain the URL right? But it will be more sophisticated because we can't find all the links with google operators like "inurl, site, link etc."

    1a. if possible I would like to track new/lost
    1b. not needed: (because I can see the linkprofile in masjestic seo) quality of the link (like pr,ct,tf and other factors)
    2. I would also like crawler anchor texts
    3. ref domains
    4. keyword rankings (daily/monthly)
    5. social activity

    I will be very very glad for your advice and ready to buy you a cup of coffe or tea because I actually am a freelancer and for my projcets it would be very helpful to have such tool (one day i will combine everything and release ultimate tool because now it's a little bit of everything everywhere). I am becoming whitehat, adding value to content, doing longterm seo but I need to watch competitors that's why I need these services.

    edit: about the question "imitate the scale of ahrefs/majestic/etc.

    I don't really need THAT BIG crawler or goal like that. I'd use it for personal projects or with my team or invite more people but I do not need that to be SO GLOBAL.

    If it's possible I would like to for example crawl (if it's easier for the bot) all the "furniture" sites or some regions like German, French, Czech etc. sites. I don't know what's easier (if it matters) to the crawler, that crawler doesn't need to crawl all the bullsh*
     
    Last edited: Oct 21, 2013