1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

looking for good robots.txt file

Discussion in 'White Hat SEO' started by dabears, Jun 21, 2012.

  1. dabears

    dabears Newbie

    Joined:
    Apr 9, 2012
    Messages:
    11
    Likes Received:
    1
    Hi everyone,

    This is a robots.txt file I got from a wso I bought called on page domination. According to the book it is "the cream of the crop" for robot files. I actually did have have it on a site earlier this year and didn't notice a big change but unfortunately that site got hacked :( and I ended up re-building the site and am now deciding if I should put this on the site again..any thoughts or does anybody know of a more powerful one?

    Thanks

    here it is:



    Sitemap: your site here ( i removed this part )

    User-agent: Mediapartners-Google
    Disallow:


    User-agent: *
    Disallow: /deal/
    Disallow: /cgi-bin/
    Disallow: /temp/
    Disallow: /wp-login.php
    Disallow: /wp-admin/
    Disallow: /wp-comments-post.php
    Disallow: /wp-commentsrss2.php


    User-agent: *
    Disallow: /*.gif$
    Disallow: /*.jpg$
    Disallow: /*.jpeg$
    Disallow: /*.png$
    Disallow: /*.zip$
    Disallow: /*.doc$
    Disallow: /*.exe$
    Disallow: /*.pdf$


    User-agent: ia_archiver
    Disallow: /
    User-agent: atSpider
    Disallow: /
    User-agent: b2w/0.1
    Disallow: /
    User-agent: BecomeBot
    Disallow: /
    User-agent: CheeseBot
    Disallow: /
    User-agent: CherryPicker
    Disallow: /
    User-agent: CopyRightCheck
    Disallow: /
    User-agent: Copernic
    Disallow: /
    User-agent: Crescent
    Disallow: /
    User-agent: DSurf
    Disallow: /
    User-agent: dumbot
    Disallow: /
    User-agent: EliteSys Entry
    Disallow: /
    User-agent: EmailCollector
    Disallow: /
    User-agent: EmailSiphon
    Disallow: /
    User-agent: EmailWolf
    Disallow: /
    User-agent: Enterprise_Search/1.0
    Disallow: /
    User-agent: Enterprise_Search
    Disallow: /
    User-agent: es
    Disallow: /
    User-agent: ExtractorPro
    Disallow: /
    User-agent: Flaming AttackBot
    Disallow: /
    User-agent: FreeFind
    Disallow: /
    User-agent: grub
    Disallow: /
    User-agent: grub-client
    Disallow: /
    User-agent: Hatena Antenna
    Disallow: /
    User-agent: Jetbot
    Disallow: /
    User-agent: Jetbot/1.0
    Disallow: /
    User-agent: larbin
    Disallow: /
    User-agent: Mail Sweeper
    Disallow: /
    User-agent: munky
    Disallow: /
    User-agent: naver
    Disallow: /
    User-agent: NetMechanic
    Disallow: /
    User-agent: Nutch
    Disallow: /
    User-agent: OmniExplorer_Bot
    Disallow: /
    User-agent: Oracle Ultra Search
    Disallow: /
    User-agent: PerMan
    Disallow: /
    User-agent: ProWebWalker
    Disallow: /
    User-agent: psbot
    Disallow: /
    User-agent: Python-urllib
    Disallow: /
    User-agent: Radiation Retriever 1.1
    Disallow: /
    User-agent: Roverbot
    Disallow: /
    User-agent: searchpreview
    Disallow: /
    User-agent: SiteSnagger
    Disallow: /
    User-agent: sootle
    Disallow: /
    User-agent: Stanford
    Disallow: /
    User-agent: URL_Spider_Pro
    Disallow: /
    User-agent: WebBandit
    Disallow: /
    User-agent: WebEmailExtrac
    Disallow: /
    User-agent: WebVac
    Disallow: /
    User-agent: WebZip
    Disallow: /
    User-agent: xGet
    Disallow: /
    User-agent: wGet
    Disallow: /
    User-agent: WebWalk
    Disallow: /
    User-agent: webvac
    Disallow: /
    User-agent: WebReaper
    Disallow: /
    User-agent: WebMirror
    Disallow: /
    User-agent: WebFetcher
    Disallow: /
    User-agent: WebCopy
    Disallow: /
    User-agent: webcopier
    Disallow: /
    User-agent: WebCatcher
    Disallow: /
    User-agent: WebBandit
    Disallow: /
    User-agent: w3mir
    Disallow: /
    User-agent: vobsub
    Disallow: /
    User-agent: Templeton
    Disallow: /
    User-agent: ssearcher100
    Disallow: /
    User-agent: SpiderBot
    Disallow: /
    User-agent: Shai'Hulud
    Disallow: /
    User-agent: PBWF
    Disallow: /
    User-agent: LightningDownload
    Disallow: /
    User-agent: KDD Exploror
    Disallow: /
    User-agent: Jeeves
    Disallow: /
    User-agent: Internet Explore
    Disallow: /
    User-agent: InfoSpiders
    Disallow: /
    User-agent: httrack
    Disallow: /
    User-agent: HavIndex
    Disallow: /
    User-agent: GetUrl
    Disallow: /
    User-agent: GetBot
    Disallow: /
    User-agent: ESIRover
    Disallow: /
    User-agent: Download Wonder
    Disallow: /
    User-agent: Collage
    Disallow: /
    User-agent: LNSpiderguy
    Disallow: /
    User-agent: Alexibot
    Disallow: /
    User-agent: Teleport
    Disallow: /
    User-agent: TeleportPro
    Disallow: /
    User-agent: Stanford Comp Sci
    Disallow: /
    User-agent: MIIxpc
    Disallow: /
    User-agent: Telesoft
    Disallow: /
    User-agent: Website Quester
    Disallow: /
    User-agent: moget/2.1
    Disallow: /
    User-agent: WebZip/4.0
    Disallow: /
    User-agent: WebStripper
    Disallow: /
    User-agent: WebSauger
    Disallow: /
    User-agent: WebCopier
    Disallow: /
    User-agent: NetAnts
    Disallow: /
    User-agent: Mister PiX
    Disallow: /
    User-agent: WebAuto
    Disallow: /
    User-agent: TheNomad
    Disallow: /
    User-agent: WWW-Collector-E
    Disallow: /
    User-agent: RMA
    Disallow: /
    User-agent: libWeb/clsHTTP
    Disallow: /
    User-agent: asterias
    Disallow: /
    User-agent: httplib
    Disallow: /
    User-agent: turingos
    Disallow: /
    User-agent: spanner
    Disallow: /
    User-agent: InfoNaviRobot
    Disallow: /
    User-agent: Harvest/1.5
    Disallow: /
    User-agent: Bullseye/1.0
    Disallow: /
    User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
    Disallow: /
    User-agent: CherryPickerSE/1.0
    Disallow: /
    User-agent: CherryPickerElite/1.0
    Disallow: /
    User-agent: WebBandit/3.50
    Disallow: /
    User-agent: NICErsPRO
    Disallow: /
    User-agent: Microsoft URL Control - 5.01.4511
    Disallow: /
    User-agent: DittoSpyder
    Disallow: /
    User-agent: Foobot
    Disallow: /
    User-agent: SpankBot
    Disallow: /
    User-agent: BotALot
    Disallow: /
    User-agent: lwp-trivial/1.34
    Disallow: /
    User-agent: lwp-trivial
    Disallow: /
    User-agent: BunnySlippers
    Disallow: /
    User-agent: Microsoft URL Control - 6.00.8169
    Disallow: /
    User-agent: URLy Warning
    Disallow: /
    User-agent: Wget/1.6
    Disallow: /
    User-agent: Wget/1.5.3
    Disallow: /
    User-agent: Wget
    Disallow: /
    User-agent: LinkWalker
    Disallow: /
    User-agent: cosmos
    Disallow: /
    User-agent: moget
    Disallow: /
    User-agent: hloader
    Disallow: /
    User-agent: URL Control
    Disallow: /
    User-agent: Zeus Link Scout
    Disallow: /
    User-agent: Zeus 32297 Webster Pro V2.9 Win32
    Disallow: /
    User-agent: Webster Pro
    Disallow: /
    User-agent: EroCrawler
    Disallow: /
    User-agent: LinkScan/8.1a Unix
    Disallow: /
    User-agent: Keyword Density/0.9
    Disallow: /
    User-agent: Kenjin Spider
    Disallow: /
    User-agent: Iron33/1.0.2
    Disallow: /
    User-agent: Bookmark search tool
    Disallow: /
    User-agent: GetRight/4.2
    Disallow: /
    User-agent: FairAd Client
    Disallow: /
    User-agent: Gaisbot
    Disallow: /
    User-agent: humanlinks
    Disallow: /
    User-agent: LinkextractorPro
    Disallow: /
    User-agent: Offline Explorer
    Disallow: /
    User-agent: Mata Hari
    Disallow: /
    User-agent: LexiBot
    Disallow: /
    User-agent: Web Image Collector
    Disallow: /
    User-agent: The Intraformant
    Disallow: /
    User-agent: True_Robot/1.0
    Disallow: /
    User-agent: True_Robot
    Disallow: /
    User-agent: BlowFish/1.0
    Disallow: /
    User-agent: JennyBot
    Disallow: /
    User-agent: MIIxpc/4.2
    Disallow: /
    User-agent: BuiltBotTough
    Disallow: /
    User-agent: ProPowerBot/2.14
    Disallow: /
    User-agent: BackDoorBot/1.0
    Disallow: /
    User-agent: toCrawl/UrlDispatcher
    Disallow: /
    User-agent: WebEnhancer
    Disallow: /
    User-agent: suzuran
    Disallow: /
    User-agent: VCI WebViewer VCI WebViewer Win32
    Disallow: /
    User-agent: VCI
    Disallow: /
    User-agent: Szukacz/1.4
    Disallow: /
    User-agent: QueryN Metasearch
    Disallow: /
    User-agent: Openfind
    Disallow: /
    User-agent: Zeus
    Disallow: /
    User-agent: RepoMonkey Bait & Tackle/v1.01
    Disallow: /
    User-agent: RepoMonkey
    Disallow: /
    User-agent: Microsoft URL Control
    Disallow: /
    User-agent: Openbot
    Disallow: /
     
  2. Nosleepever

    Nosleepever Registered Member

    Joined:
    Jun 23, 2010
    Messages:
    97
    Likes Received:
    17
    Bump

    I am looking for a good one also.