Help with robots.txt and sitemap.xml

[Deleted member 640294]

Hi guys.

For a while now I've been messing with google webmasters, and it always shows that my robots txt is blocking my sitemap. I've tried editing it, but it keeps going back to this default after a few hours.

 Sitemap: websitedotcom/sitemapdotxml
User-agent: *
Disallow: /

I've also deleted the robots file from my directory and re-created it, and the same result happens.

Also with my sitemap xml file, webmasters has been saying that there is an xml tag missing. The parent tag is urlset and the tag is url. This is just a small snippet of my xml file.

<?xml version="1.0" encoding="UTF-8"?>
<urlset
<url>
  <loc>hyperlinkwebsitedotcom/indexdothtml</loc>
  <lastmod>2014-02-02T07:00:01+00:00</lastmod>
  <changefreq>weekly</changefreq>
  <priority>0.80</priority>
</url>
</urlset>

It also comes with a heading which I have also tested by deleting a few times. So far nothing has changed.

Any help would is much appreciated.

 

[kalasnikof1]

1.
plugin mess up
2.
admin panel >> Settings >> Reading
before the Save Changes button is a checkbox with the label Search Engine Visibility make sure this is checked.

 

[Deleted member 640294]

By "admin panel" I'm assuming you mean google analytics?

If so I see just "account", "property" and "view" settings. Nothing that is just "settings" and nothing can be found to do with "Reading".

Im not finding an "admin panel" in google webmasters, the only "settings" I see are "site settings" which dont seem to contain anything to do with "reading". Could you please be further exact?

 

[kvchosting]

Remove it completely from GWT then create your own robots.txt, this would be better therefore you will not confuse between GWT and robots.txt file

 

[zee007]

Is there a reason you are creating a robots.txt file? Unless you have pages you want to actually block, there is no need to create a robots.txt file in the first place! An XML sitemap on the other hand is very valuable. You should create an XML sitemap and upload it to the root folder of your website. Then submit it via WMT.

 

[evilclown]

Your sitemap is incomplete, here use this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>stuff goes her</url>

</urlset>

If you are not blocking pages, delete your robot.txt file.

 

[GloCk99]

The solution to this is actually fairly simple. There is a lag in what google webmaster tools shows as your robots.txt and what you actually have as a robots.txt setup. It can take up to 2 days for the correct one to show in webmaster tools.

Set your robots.txt in your root directory sitename.com/robots.txt . Once you've confirmed it's correct by visiting the url, then just wait! Stop changing the robots.txt frequently as you'll keep hitting this lag. Leave it for a couple of days and go back in WMT and you'll see the correct one being displayed.

Good luck.

@http://www.blackhatworld.com/blackhat-seo/members/373883-kvchosting.html if you don't know what you're talking about, then don't try and give advice.

 

[CredibleZephyre]

Is there a reason you are creating a robots.txt file? Unless you have pages you want to actually block, there is no need to create a robots.txt file in the first place! An XML sitemap on the other hand is very valuable. You should create an XML sitemap and upload it to the root folder of your website. Then submit it via WMT.

I'm not sure he did this on purpose but in his first example he's disallowing his entire site. If that's what he is trying to do then yes, he would need a robots.txt file

 

[GloCk99]

I'm not sure he did this on purpose but in his first example he's disallowing his entire site. If that's what he is trying to do then yes, he would need a robots.txt file

That's not what he's using on his site. The example he's given is the WMT default. There are numerous reasons why someone would want to optimise a robots.txt file.

 

[Deleted member 640294]

Your sitemap is incomplete, here use this:

Thanks evilclown, that got my site map up for me.

in his first example he's disallowing his entire site.

When I originally purchased my domain, that's what the robots file looked like when I initially opened it the first time after publishing it. Im trying to modify it so that 2 subdomains I'm creating for more pages to add to my website (hosted by hostgator and created with weebly sitebuilder. Max page limit is 6, but with unlimited subdomains I can very easily work around this issue) get blocked. In the footer section I have my site name which links back to my original domain. I want those links blocked because I don't want search engines to see those as unnecessary links. This is just 1 subdomain that I'm trying to disallow, and my sitemap file that I'm trying to allow (but I don't think will be necessary anymore since my sitemap is seen in GWT), I can remove it if you guys think it is irrelevant. Also included, I've found a list of bad robots that either seem malicious, or have already been disallowed by many other websites.

User-agent: *
Disallow: hyperlinksubdomainDOTsiteDOTcom/

User-agent: *
Allow: hyperlinkdomainDOTcom/sitemapDOTxml


###
#Unsafe robots to keep away
###
User-agent: Aqua_Products
Disallow: /
User-agent: asterias
Disallow: /