Per

<verbatim>
On Dec 2, 2009, at 10:52 AM, Daniel Clark wrote:

> For a while it's seemed odd to me that google hadn't updated its
> indexes of a few sites, so I took a look and noticed the obvious -
> robots.txt was disallowing all crawlers:
>
> (from main.c)
>
>  /* Prevent robots from indexing this site.
>  */
>  if( strcmp(g.zPath, "robots.txt")==0 ){
>    cgi_set_content_type("text/plain");
>    @ User-agent: *
>    @ Disallow: /
>    cgi_reply();
>    exit(0);
>  }
>
> As far as I can tell there isn't a way to disable or tune this from
> the fossil level; if there isn't interest in changing this I'm sure I
> can just redirect via apache to some actual file, but IMHO it would be
> good to be able to easily make fossil projects searchable (perhaps even
> have this be the default), esp. since at the moment the only reason that
> say is searchable is because the robots.txt file happens
> to be at:
>
>
>
> (eg a random apache configuration choice to have rather ugly URLs with
> "index.html" in all of them.)

Note that does not use apache.  The
redirect occurs within fossil itself.

I suppose that since the existing "robots.txt" is essentially a no-op,
we might as well remove it.
</verbatim>