1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt problem...

Discussion in 'BlackHat Lounge' started by Kidax, Oct 6, 2016.

  1. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Hi,

    I downloaded a plugin "Virtual robots.txt" for Wordpress. The plugin change my robots.txt but now I can't edit it. So I remove it. But my robot is always the same. I tried to download some others plugins but it override everything. When I upload a physical robot.txt on my root it override too... I'm on the problem since 5 hours, I'm tired. If my problem is resolve, I don't download plugins anymore... Everytime I've got problems.

    Someone can help me please ?

    Code:
    # This virtual robots.txt file was created by the Virtual Robots.txt WordPress plugin: https://www.wordpress.org/plugins/pc-robotstxt/
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    Disallow: /wp-includes/
    Allow: /wp-includes/js/
    Allow: /wp-includes/images/
    Disallow: /trackback/
    Disallow: /wp-login.php
    Disallow: /wp-register.php
     
  2. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    24
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    Do you have any other plugin that might be caching the robots.txt file?
     
  3. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    I don't think... I disabled all my plugins for a test and nothing change.
     
  4. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    24
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    Where is the website hosted?
     
  5. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Easyname.com, it's an issue with my host ? I use Cloudflare too.
     
  6. nahid133

    nahid133 Senior Member

    Joined:
    May 7, 2011
    Messages:
    1,036
    Likes Received:
    288
    Gender:
    Male
    Occupation:
    finding intruders
    Location:
    192.168.0.1
    Home Page:
    Try to edit your robot.txt file. You can do it through Yoast seo plugins. Google for more help.
     
  7. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    24
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    You'd try contacting your host and tell them about the problem. Maybe they're automatically caching your content. Or, as @nahid133 suggested try using Yoast's SEO Plugin, if you have it installed to edit the file before you contact your host.
     
  8. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    I tried... A lot... There's no robots.txt from "Virtual robots.txt" plugin...
     
  9. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    I had just created my robots.txt. Do you think it's possible to caching instantly ?
     
  10. nahid133

    nahid133 Senior Member

    Joined:
    May 7, 2011
    Messages:
    1,036
    Likes Received:
    288
    Gender:
    Male
    Occupation:
    finding intruders
    Location:
    192.168.0.1
    Home Page:
    Forget "Virtual robots.txt" plugin... as u already uninstalled it . try to find your robot.txt file
     
  11. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Guys thank you... That was cache... I'm going on Cloudflare, I make a purge and my robots.txt have changed !!! Thank you very much. I was desperate.
     
    • Thanks Thanks x 1
  12. vistatech

    vistatech Newbie

    Joined:
    May 31, 2014
    Messages:
    24
    Likes Received:
    2
    Gender:
    Male
    Occupation:
    Medico
    Location:
    YouTube :)
    Thought so.
     
  13. AstTrailerParkSupervisor

    AstTrailerParkSupervisor Newbie

    Joined:
    Sep 3, 2016
    Messages:
    26
    Likes Received:
    10
    Gender:
    Male
    Just curious, wouldn't it be easier to just manually edit the robots.txt and upload it without using a plugin to do so?
     
  14. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Yes totally easier. But I couldn't remember that was so easy.... It's been a long time since I've made a website.
     
  15. AstTrailerParkSupervisor

    AstTrailerParkSupervisor Newbie

    Joined:
    Sep 3, 2016
    Messages:
    26
    Likes Received:
    10
    Gender:
    Male
    Ok cool, I'm no genius when it comes to webmastery either, just wanted to see if I was missing out on something lol
     
  16. umerjutt00

    umerjutt00 Jr. VIP Jr. VIP

    Joined:
    Oct 28, 2011
    Messages:
    3,822
    Likes Received:
    2,061
    Occupation:
    Ninja
    Exactly. Next time when you make a change in your robot.txt or even css file, just remember to purge the cache in CloudFlare. Once I made some changes to my CSS file and it was still showing the old one. Took me few hours to realize that it was a cache problem at CF.
     
  17. Kidax

    Kidax Newbie

    Joined:
    Dec 31, 2014
    Messages:
    43
    Likes Received:
    13
    Haha no, you were right! But when you start to download a plugin, you think you need an other plugin for everything lol

    Next time it will be the first thing I will think... I searched everywhere and I was so tired. That was exhausting! But it's ok, all my mistakes learn me to search and ask for help. I'm not familiar with that.