This technical article was first published over 2 years ago and may discuss features and approaches that are no longer relevant for the current version of the platform.
David Knipe put together a really nice module for EPiServer CMS 6R2 called EPiRobots which allowed Site Administrators to manage and update a site’s robots.txt content. Unfortunately the module doesn’t work with CMS 7 🙁
I asked Dave about his plans for a version compatible with CMS 7 and there is apparently some work/ideas in the pipeline. However that news did not satisfy my IMMEDIATE requirement! Dave also mentioned that I could ‘do whatever I wanted with the source code’. So I’ve forked it!
I’ve upgraded the forked version to work against CMS 7. The main change was to remove an initialisation piece that added an explicit handler for the Robots.txt path through the EPiServer UrlRewriting API (which has been overhauled for CMS 7). The robots.txt content is now served from a standard MVC controller with a route being configured in an EPiServer initialization module. I’ve also rebuilt the Admin user interface using MVC and explicitly included the views as content within the NuGet package rather than serving them from the .dll. This is just my personal preference. Leveraging NuGet and the simplicity of the solution wins for me! It also means you can change the view look and feel if you so wish.
The source code is available at https://github.com/markeverard/POSSIBLE.RobotsTxtHandler
A package (now renamed from EPiRobots) will be available in the EPiServer Nuget Feed (as soon as its been approved) – http://nuget.episerver.com/ – search for POSSIBLE.RobotsTxtHandler