Robots.txt and Indexing problem?!

D

devbrah

Junior Member
Joined
Oct 10, 2011
Messages
111
Reaction score
15
Hi guys,

So i've been trying to rank a MNS for the last month and a bit, and my webmaster tools has been set up the entire time.

This is what my robots.txt looks like currently:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

As you can see, it's a wordpress site.

My rankings have been dropped, and stayed down, and my impressions have dropped to <10 from an average of around 20/day.

This I would attribute to not doing much with the site lately as i've been really busy, but when I look at my keywords, this is all it's giving me:


1. disallow
2. admin
3. agent
4. user

What's going on here? It's like it's not even indexing any of my content at all.

Can anyone provide any insight?

Thanks!!
 
Advertise on BHW
User-agent: * pointing to the web browser web crawler robot

Disallow: /wp-admin/
Disallow: /wp-includes/


in Disallow means web browser crawler robot will not crawl on that folder because that folder is important for wordpress if browser crawler robot will index it your website have the possibility to have


Hope that makes sense
 
interesting. look like robot.txt files cause problem for lots people on last days.
 
I don't want to give out my URL as i'm new to ranking niche sites, and don't want someone swooping in and outranking me for my target keyphrases haha.

Anyone have any ideas as to why this is happening?
 
Is your robots.txt file being delivered as a text file?
 
Last edited:
USe this Robots.txt example
Code: 
Sitemap: http://www.yourwebsite.com/sitemap.xml
User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
User-agent: Googlebot
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /*?*
Disallow: /z/
Disallow: /wp-*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
 
Knoxgates said:
USe this Robots.txt example

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
User-agent: Googlebot
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /*?*
Disallow: /z/
Disallow: /wp-*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$[/CODE]
Click to expand...

... i'm not an expert on the subject, but this looks like it's disallowing pretty much every possible file extension I would want to use.

Am I wrong?
 
nobody can help me with this?
 
orlandolongwood said:
Is your robots.txt file being delivered as a text file?
Click to expand...

Google webmaster tools tells me it's good.

This is what's currently in there:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Any ideas?

I think this is hurting my ranks :(

My first niche site isn't going as expected haha.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top