Robots.txt problem...

Discussion in 'BlackHat Lounge' started by Kidax, Oct 6, 2016.

  1. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Hi,

    I downloaded a plugin "Virtual robots.txt" for Wordpress. The plugin change my robots.txt but now I can't edit it. So I remove it. But my robot is always the same. I tried to download some others plugins but it override everything. When I upload a physical robot.txt on my root it override too... I'm on the problem since 5 hours, I'm tired. If my problem is resolve, I don't download plugins anymore... Everytime I've got problems.

    Someone can help me please ?

    Code:
    # This virtual robots.txt file was created by the Virtual Robots.txt WordPress plugin: https://www.wordpress.org/plugins/pc-robotstxt/
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    Disallow: /wp-includes/
    Allow: /wp-includes/js/
    Allow: /wp-includes/images/
    Disallow: /trackback/
    Disallow: /wp-login.php
    Disallow: /wp-register.php
     
  2. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    31
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    Do you have any other plugin that might be caching the robots.txt file?
     
  3. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    I don't think... I disabled all my plugins for a test and nothing change.
     
  4. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    31
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    Where is the website hosted?
     
  5. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Easyname.com, it's an issue with my host ? I use Cloudflare too.
     
  6. nahid133

    nahid133 Supreme Member

    Joined:
    May 7, 2011
    Messages:
    1,295
    Likes Received:
    395
    Gender:
    Male
    Occupation:
    finding intruders
    Location:
    192.168.0.1
    Home Page:
    Try to edit your robot.txt file. You can do it through Yoast seo plugins. Google for more help.
     
  7. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    31
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    You'd try contacting your host and tell them about the problem. Maybe they're automatically caching your content. Or, as @nahid133 suggested try using Yoast's SEO Plugin, if you have it installed to edit the file before you contact your host.
     
  8. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    I tried... A lot... There's no robots.txt from "Virtual robots.txt" plugin...
     
  9. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    I had just created my robots.txt. Do you think it's possible to caching instantly ?
     
  10. nahid133

    nahid133 Supreme Member

    Joined:
    May 7, 2011
    Messages:
    1,295
    Likes Received:
    395
    Gender:
    Male
    Occupation:
    finding intruders
    Location:
    192.168.0.1
    Home Page:
    Forget "Virtual robots.txt" plugin... as u already uninstalled it . try to find your robot.txt file
     
  11. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Guys thank you... That was cache... I'm going on Cloudflare, I make a purge and my robots.txt have changed !!! Thank you very much. I was desperate.
     
    • Thanks Thanks x 1
  12. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    31
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    Thought so.
     
  13. AstTrailerParkSupervisor

    AstTrailerParkSupervisor Newbie

    Joined:
    Sep 3, 2016
    Messages:
    26
    Likes Received:
    10
    Gender:
    Male
    Just curious, wouldn't it be easier to just manually edit the robots.txt and upload it without using a plugin to do so?
     
  14. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Yes totally easier. But I couldn't remember that was so easy.... It's been a long time since I've made a website.
     
  15. AstTrailerParkSupervisor

    AstTrailerParkSupervisor Newbie

    Joined:
    Sep 3, 2016
    Messages:
    26
    Likes Received:
    10
    Gender:
    Male
    Ok cool, I'm no genius when it comes to webmastery either, just wanted to see if I was missing out on something lol
     
  16. umerjutt00

    umerjutt00 Jr. VIP Jr. VIP

    Joined:
    Oct 28, 2011
    Messages:
    3,980
    Likes Received:
    2,220
    Occupation:
    Ninja
    Exactly. Next time when you make a change in your robot.txt or even css file, just remember to purge the cache in CloudFlare. Once I made some changes to my CSS file and it was still showing the old one. Took me few hours to realize that it was a cache problem at CF.
     
  17. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Haha no, you were right! But when you start to download a plugin, you think you need an other plugin for everything lol

    Next time it will be the first thing I will think... I searched everywhere and I was so tired. That was exhausting! But it's ok, all my mistakes learn me to search and ask for help. I'm not familiar with that.