D 2010-01-15T17:21:45
J comment Per\s[http://thread.gmane.org/gmane.comp.version-control.fossil-scm.user/1489]:\r\n\r\n<verbatim>\r\nOn\sDec\s2,\s2009,\sat\s10:52\sAM,\sDaniel\sClark\swrote:\r\n\r\n>\sFor\sa\swhile\sit's\sseemed\sodd\sto\sme\sthat\sgoogle\shadn't\supdated\sits\s\s\r\n>\sindexes\sof\sa\sfew\ssites,\sso\sI\stook\sa\slook\sand\snoticed\sthe\sobvious\s-\r\n>\srobots.txt\swas\sdisallowing\sall\scrawlers:\r\n>\r\n>\s(from\smain.c)\r\n>\r\n>\s\s/*\sPrevent\srobots\sfrom\sindexing\sthis\ssite.\r\n>\s\s*/\r\n>\s\sif(\sstrcmp(g.zPath,\s"robots.txt")==0\s){\r\n>\s\s\s\scgi_set_content_type("text/plain");\r\n>\s\s\s\s@\sUser-agent:\s*\r\n>\s\s\s\s@\sDisallow:\s/\r\n>\s\s\s\scgi_reply();\r\n>\s\s\s\sexit(0);\r\n>\s\s}\r\n>\r\n>\sAs\sfar\sas\sI\scan\stell\sthere\sisn't\sa\sway\sto\sdisable\sor\stune\sthis\sfrom\s\s\r\n>\sthe\sfossil\slevel;\sif\sthere\sisn't\sinterest\sin\schanging\sthis\sI'm\ssure\sI\r\n>\scan\sjust\sredirect\svia\sapache\sto\ssome\sactual\sfile,\sbut\sIMHO\sit\swould\sbe\s\s\r\n>\sgood\sto\sbe\sable\sto\seasily\smake\sfossil\sprojects\ssearchable\s(perhaps\seven\s\s\r\n>\shave\sthis\sbe\sthe\sdefault),\sesp.\ssince\sat\sthe\smoment\sthe\sonly\sreason\sthat\s\s\r\n>\ssay\sfossil-scm.org\sis\ssearchable\sis\sbecause\sthe\srobots.txt\sfile\shappens\r\n>\sto\sbe\sat:\r\n>\r\n>\shttp://fossil-scm.org/index.html/robots.txt\r\n>\r\n>\s(eg\sa\srandom\sapache\sconfiguration\schoice\sto\shave\srather\sugly\sURLs\swith\r\n>\s"index.html"\sin\sall\sof\sthem.)\r\n\r\nNote\sthat\shttp://www.fossil-scm.org/\sdoes\snot\suse\sapache.\s\sThe\s\s\r\nredirect\soccurs\swithin\sfossil\sitself.\r\n\r\nI\ssuppose\sthat\ssince\sthe\sexisting\s"robots.txt"\sis\sessentially\sa\sno-op,\s\s\r\nwe\smight\sas\swell\sremove\sit.\r\n</verbatim>
J foundin 20100112141013
J private_contact ace2a438fb254e5d21ba2ed1295b2da8392d271c
J severity Minor
J status Open
J title Disable\sbuilt-in\srobots.txt\sfile
J type Code_Defect
K 03c9f4deef2d19f55945415cd1d048b4d730cf8c
U anonymous
Z 4830c8977cd1b93c0eb229503677fded