Robot.txt still used?

Create&Crwal · Jul 10, 2024

Robots.txt is useful from my perspective as a website owner. It helps to instruct Google and other search engines on what to crawl and what not to crawl.
Additionally, it's effective for hiding-

Admin pages.
Login pages

DreamChaserr · Jul 10, 2024

Yes, of course is still used.

MrFarmer · Jul 10, 2024

Yes. it is still required by Google.

shipsocker · Jul 10, 2024

yes. WP automatically generates this file and I'm sure a lot of other CMSs, so I'd say that we can estimate that at least half of the sites on the web use it.

inertia333 · Jul 10, 2024

Create&Crwal said:
Robots.txt is useful from my perspective as a website owner. It helps to instruct Google and other search engines on what to crawl and what not to crawl.
Additionally, it's effective for hiding-

Admin pages.
Login pages
View attachment 359264

Yeah, it's still useful for instructing search engines.

helping to protect sensitive information and improve SEO.

Create&Crwal · Jul 10, 2024

shipsocker said:
yes. WP automatically generates this file and I'm sure a lot of other CMSs, so I'd say that we can estimate that at least half of the sites on the web use it.

Yes, but make sure to check or add few you not want to seen.

Robot.txt still used?

Create&Crwal

BANNED

DreamChaserr

Junior Member

MrFarmer

BANNED

shipsocker

Junior Member

inertia333

Registered Member

Create&Crwal

BANNED

Main Menu

Marketplace

Making Money

BlackHat World