1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Prevent Pages from showin up in the SERPs

Discussion in 'Black Hat SEO' started by ichiban, Jul 13, 2011.

  1. ichiban

    ichiban Junior Member

    Joined:
    Jan 31, 2011
    Messages:
    180
    Likes Received:
    10
    I have a few pages that show in the results that I would rather not.

    Namely a "admin-ajax.php" that appears as "-1" - I downloaded the robot meta plugin and made sure to "no follow no index" all admin pages. I ust did this two days ago so It might be a matter of g00gle not crawling the site to update cache.

    However, I just inserted a download link via download manager plugin less than 48 hours ago & It already shows up in the search results. How do I prevent this download link from showing up in the serps?
     
  2. altaris

    altaris Registered Member

    Joined:
    Apr 24, 2011
    Messages:
    67
    Likes Received:
    5
    Additionally to noindex header you can block search engines in robot.txt
     
  3. ichiban

    ichiban Junior Member

    Joined:
    Jan 31, 2011
    Messages:
    180
    Likes Received:
    10

    thats the thing! I cant find my robot.txt file. I go into my file manager in Cpanel then search for "robot.txt" in "all files". It says no such file exists.

    I dont know what to do?
     
  4. sircoldfire

    sircoldfire Junior Member

    Joined:
    Jul 2, 2011
    Messages:
    104
    Likes Received:
    27
    Occupation:
    motha fu¬Ękin hu$tla
    Location:
    bank teller window cashin checks
    go gle up how to make a file ... i know you put one where ever you want and its top down from there so if put in the base of your admin folder tree should keep the bots out... also if its possible you might consider moving the admin folder if it was indexed for safety sake in addition to the no robots .. other option is to .htaccess a password onto the folder director for the admin stuff .. actualy the .htaccess password ontop of the normal admin login is a really good idea
     
  5. altaris

    altaris Registered Member

    Joined:
    Apr 24, 2011
    Messages:
    67
    Likes Received:
    5
    Create it, it is a normal txt file. Check syntax or use an online tool, you only need some allow or disallow and the url.
     
    • Thanks Thanks x 1
  6. ichiban

    ichiban Junior Member

    Joined:
    Jan 31, 2011
    Messages:
    180
    Likes Received:
    10

    Ok, so once I create it where do I put it? I have an addon domain from host gator. Should I put the robot.txt file in my websites root folder or should it go elsewhere?

    Furthermore, what code do I need to write in it to prevent all admin pages from being crawled and to prevent my download link from being crawled as well?


    Thanks!
     
  7. Autumn

    Autumn Elite Member

    Joined:
    Nov 18, 2010
    Messages:
    2,197
    Likes Received:
    3,041
    Occupation:
    I figure out ways to make money online and then au
    Location:
    Spamville
    Say you want to disallow spiders from indexing http://yourdomain.com/admin-ajax.php and http://yourdomain.com/download.php:

    Make a plain text file called robots.txt with these contents:

    Code:
    User-agent: *
    Disallow: /download.php
    
    User-agent: *
    Disallow: /admin-ajax.php
    
    If you wanted to block a whole directory, use something like:

    Code:
    User-agent: *
    Disallow: /admin/
    
    Then upload it to the root dir for that particular domain on your hosting; from memory on Hostgator it's probably /addons/yourdomain.com/public_html. The robots.txt will be retrieved next time Googlebot comes around. From my experience, it then takes a few days for the urls to be dropped from the index, if they have already been indexed.
     
    • Thanks Thanks x 1
  8. Monrox

    Monrox Power Member

    Joined:
    Apr 9, 2010
    Messages:
    615
    Likes Received:
    579
    You should be aware that doing it via robots.txt tells everyone where to look for the things you want to hide. Most search engines honor the instructions but anyone with a half brain can type 'yourdomain.com/robots.txt' and be presented with the info.

    Have a look at: www.bing.com/robots.txt
     
  9. ichiban

    ichiban Junior Member

    Joined:
    Jan 31, 2011
    Messages:
    180
    Likes Received:
    10

    Fair enough...in that case what should I do?

    As you can see I'm sure as shit not a programmer & I barely know coding. All I want to do is prevent any back-end admin pages from being crawled & especially to prevent my new download link from showing up in the serps.

    Any suggestion Monrox?
     
  10. oliva999

    oliva999 Power Member

    Joined:
    Feb 6, 2010
    Messages:
    572
    Likes Received:
    69
    Install plugin PC robots.txt
     
  11. Monrox

    Monrox Power Member

    Joined:
    Apr 9, 2010
    Messages:
    615
    Likes Received:
    579
    Every advice from the other posters above is sound, I was merely showing you a weakness. Short of using a notification to alert you when a user is entitled to a download and you emailing him the stuff, there is no 100% protection.

    For robots.txt you have to turn off directory indexing on your server as well. This way if you are hiding yourdomain.com/hidden_stuff, and if I go there I will get an error page and not a list to all the pages and files inside.

    For noindex metatags, load the page you want to hide and see whether the syntax in the head part of the source code is correct, something like:
    <meta name="robots" content="noindex, noarchive, nofollow">

    If you have the metatag correctly, search engines are really trying hard not be indexing.