1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Block SEMrush

Discussion in 'Black Hat SEO' started by choppa2choppa, Apr 19, 2016.

  1. choppa2choppa

    choppa2choppa Registered Member

    Joined:
    Feb 22, 2016
    Messages:
    88
    Likes Received:
    11
    I am currently adding some adwords ad but I know that SEMrush will find them soon and allow competitors to copy my ads and keyword etc. Is there a way to block SEMrush from spying all my ads out and allowing competitors to copy?

    Also does the SEMrush bot go on to websites and follow the backlink to the ad or does the bot monitor the ads that show up on google search?
     
  2. roadhamster

    roadhamster Regular Member

    Joined:
    Mar 12, 2012
    Messages:
    335
    Likes Received:
    242
    Check here on how to block the semrush bot: https://www.semrush.com/bot/

    If you're not confident with that you can block it with .htaccess file:

    BrowserMatchNoCase SemrushBot bad_bot
    Order Deny,Allow
    Deny from env=bad_bot

    An other option is to block it with PHP:
    PHP:
    <?php
    if(preg_match('/SemrushBot/i',$_SERVER['HTTP_USER_AGENT']))
        {
        
    header('HTTP/1.0 403 Forbidden');
        echo 
    "Yippie ka yee motherf#cker";
        exit();
        }
    ?>
     
  3. TyphoonAgency

    TyphoonAgency Regular Member

    Joined:
    Oct 7, 2010
    Messages:
    280
    Likes Received:
    99
    Occupation:
    Providing professional grade white label SEO servi
    Location:
    Support Ticket System
    I know you can block semrush bot with a plugin called linkprivacy. All it does is stop tools from crawling your website. Not sure if this will help you with adwords or not.
     
  4. choppa2choppa

    choppa2choppa Registered Member

    Joined:
    Feb 22, 2016
    Messages:
    88
    Likes Received:
    11
    Thanks for the help guys I have blocked it using robots.txt which looks like this below would this be ok?

    User-Agent: *

    User-agent: SemrushBot
    Disallow: /
     
  5. CallmeReaz

    CallmeReaz Junior Member

    Joined:
    Mar 18, 2014
    Messages:
    193
    Likes Received:
    38
    It won't, and this issue is a huge one for those of us in the lead generation industry.

    To be honest, if you're using Adwords right now and are looking to stop competitors clicking adverts and discovering how you earn your money follow these rules:

    1. Block your site completely from Google by using robots.txt + meta tags - This is especially true if certain pages / text on your site could identify what affiliate / lead gen network you're using like custom privacy pages or terms and conditions that contain certain statements required by the merchant.

    2. Try to steer clear of head terms - Head terms convert terribly and are typically the first port of call for click happy newbies and "SEO professionals" looking to scope the competition.

    3. Use a separate website for testing - Your testing website will almost always be bidding on broad match keywords, hence it's often the one that will be picked up by SEO experts, brand managers and other affiliates. It's also the one people will probably test Spyfu against to try and figure out how your getting your conversions. By only testing using this domain you'll always be throwing people off the scent by bidding on useless keywords. When you've found a profitable subset of keywords, add them to your negative list for the testing site and create a new campaign for your main conversion site - and always try to either use phrase match, exact match or try to bid just enough on a broad match to make a profit. You want to minimize your exposure to having the main sales website discovered, especially if you've worked hard to find the perfect design to promote your conversion rate.

    4. If you're in a really competitive niche, switch up the sales site frequently - This is especially true to certain financial and compensation niches, competitors especially other affiliates will use Spyfu to try and figure out what keywords you're bidding on. By changing your sites every 3 months you'll keep them off the scent.

    5. Private whois - Always use a private whois. Competitors will try to look for the links between brands, websites and company names.
     
    • Thanks Thanks x 2
    Last edited: Apr 20, 2016
  6. Cha Wong

    Cha Wong Regular Member

    Joined:
    Feb 1, 2016
    Messages:
    230
    Likes Received:
    10
    Occupation:
    SEO Expert & Social Media Marketer (Beginner)
    Location:
    Singapore
    I think robots.txt will work if you use it
     
  7. ThopHayt

    ThopHayt Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 25, 2011
    Messages:
    6,355
    Likes Received:
    2,304
    Those robot files are only a REQUEST to not be crawled by a bot. If SemRush wanted to ignore that request and crawl it anyway it could. And they could also run, un-branded bots. They may not do this... but I don't see why not.
     
  8. choppa2choppa

    choppa2choppa Registered Member

    Joined:
    Feb 22, 2016
    Messages:
    88
    Likes Received:
    11
    I have used robots and going to check in 2-3 weeks to see if the site is still listed in SEMrush. Also i'm guessing they would need to play by the rules if they have posted this https://www.semrush.com/bot/ if they ignore this would this be bordering illegal for a big legitimate company like them to be ignoring the bot disallow?