robots.txt

Hi All, I have seen 2 methods of blocking bots through htaccess floating around and cannot see the operating difference. Can someone please explain the difference? If you know of what is best practice that be great too... Im using wordpress themes if this make any difference. First and...

Here is a question for you guys: If I add following syntax: User-agent: * Disallow: / This means it will disallow all the bots to crawl any of my page. Right? But, would it also stop crawler from crawling my root page? Or crawler can still crawl my root page? Looking forward for great views.

A couple of days before the new year we sent live our new theme design on our Shopify store that we, which has a lot of different features to our last one, the main one being that we have a new product filter by tags sidebar that when used creates new URL paths. A week after the site launched...

Hi guys, I am facing some issues with indexing of my site. In Google webmaster, robots.txt fetch is showing a yellow sign instead of green dot. I have researched on this and created a new robot.txt through yoast SEO plugin. Still there is a yellow sign in front of robots.txt fetch. Please check...

Can anyone solve this mystery? I work on a site that has lots of good content, and also a directory. The directory pages could be classed as thin content, so they have had both a noindex directive placed on them, and been noindexed via robots.txt. Those have been in place for 18 months. The...

I'm trying to scrape or harvest .com.au websites that have their robots txt file set to block everything - Disallow: / I know these sites still show up in the search results with "A description for this result is not available because of this site's robots.txt" in their organic search results...

Can anyone tell what i need to do ? here is a message which I got domain dot com: Googlebot can't access your site Feb 8, 2014 Over the last 24 hours, Googlebot encountered 14 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we...

Hi guys. For a while now I've been messing with google webmasters, and it always shows that my robots txt is blocking my sitemap. I've tried editing it, but it keeps going back to this default after a few hours. Sitemap: websitedotcom/sitemapdotxml User-agent: * Disallow: / I've also deleted...

Hey guys, I have a plan but I don't know how safe it will be. I have a blog with adsense running on it but I want to implement a image section to the site. The problem is that it's of course going to be duplicate content and I run the risk of getting penalties. However, I plan to start a...

Hi I need help with this, currently my Robots.txt file is setup in this order as i dont want my website to be crawled in other directories other than the important ones? User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /Admin/ Disallow: /images/ Disallow: /Account/ Disallow: /bin/...

I find some site use robot.txt to block google search, then , an idea came up. I just copy the content to my site and that is the original ones. but , I search the content in google and the original site still got #1 in google. so,The robots.txt do not works?

Happen to face Google adsense crawl error, found the fix. It says, to make adsense to crawl your pages and show targetted ads, add these lines on top of robots.txt. User-agent: Mediapartners-Google Disallow: But my robots.txt file has already had this lines User-agent: Mediapartners-Google*...

Hello I am quite a newbie but I am encountering troubles with being indexed with search engines. My robots.txt file shows this /sitemap.xml.gz Is this why I am having trouble. And if so how do i fix this. I am using a purchased theme called wp-amazillionaire. Any help would be great. Thanks...

This is my first thread anyway and I thought to inform you a small tip. Before creating any Web2.0, make sure you check the robots.txt. There are many sites that disable Google crawling to avoid spam. For example - Jimdo and Onsugar. They allow Google crawling only when you upgrade your account...

I have affiliate links on my site pages and don't want Google to index them so please tell me how to set no-index for a external link on a page.

User-agent: * Disallow: /*? Disallow: *.swf$ Disallow: *.js$ Disallow: /admin/ Disallow: /xml/ In one of the site i have seen this kind of robots.txt file. can some one tell me what does this mean. As i have less knowledge in Robots.txt so to get more knowledge about this i want to know.

I have put together a robots.txt file for eCommerce. This file has been specifically designed for Magento, but the principles remain the same across eCommerce sites. Before you just upload this to your site make sure you do you research. User-agent: * Disallow: /admin/ Disallow: /app/ Disallow...

I have a blogger blog. I want to add some internal links. Lets say that I am writing posts about types of radiation, and I want to give add the link of same domain page "radiation" to this post (i.e. 1 post has other post link), like most of the wiki-pages have. Will it hurt my SEO (multiple...

Hi there:) I have a newbie question I am tired to search what i am doing wrongwith my robot txt :confused: Trying to set up my robots.txt i installed pluggin in my WP but each of these pluggins seems to don't work for me when i check for mywpsite/robots.txt i only got a 404 not fund I aslo...

Hi all, I've been searching on this topic but can't really seem to find a decent answer. I want to hide my wordpress plugin page from being indexed. I also have a few pages I don't want indexed, like a download page and some pages with content for certain people only, and I don't want them...